openvibers/openvibe.host Commits
ovhost alerts relay (roadmap WS-H task 11): the alerts firing in the host's Prometheus now reach the operator. Every two minutes (openvibe-alerts.timer) it reads /api/v1/alerts, keeps the firing ones, and sends the complete set to Network's POST /internal/operator/alerts with Host's token (network.operator.alert; Contracts v0.62.0, pinned here from v0.58.0). Network pages the owner when an alert opens, daily while open, and when it resolves. Severity page maps to critical, ticket to warning. Only the name, severity, annotations, service label and start time leave the host, and a Prometheus that does not answer sends nothing (an empty set would resolve everything). openvibe_alert_relay.prom and the OpenVibeAlertRelayFailing rule record relay failures. docs/operator-alerts.md; test/alerts.test.js
Changes
@@@ -1,6 +1,8 @@# OpenVibe alert rules (roadmap Track O), loaded by the host's Prometheus (rule_files in the config--# scripts/prometheus-config.js renders). Firing alerts are read at GET 127.0.0.1:9090/api/v1/alerts;--# no notifier is configured yet.++# scripts/prometheus-config.js renders). Firing alerts are read at GET 127.0.0.1:9090/api/v1/alerts by++# `ovhost alerts relay` (openvibe-alerts.timer, every 2 minutes; lib/alerts.js), which sends them to++# OpenVibe.Network: severity page pages the operator at once (critical), ticket at high priority, and each++# alert is reminded daily while open and announced when it resolves (docs/operator-alerts.md).groups:- name: openviberules:@@@ -77,3 +79,11 @@ groups:labels: { severity: ticket }annotations:summary: "No passing Tools job proof for more than 13 hours (two runs)"++ # The relay itself (openvibe-alerts.timer; WS-H task 11). It cannot page while it is broken; it pages++ # this as soon as a run gets through again, so an outage of the pager is never silent afterwards.++ - alert: OpenVibeAlertRelayFailing++ expr: time() - openvibe_alert_relay_last_success_timestamp_seconds > 15 * 60++ for: 5m++ labels: { severity: page }++ annotations:++ summary: "Operator alerts were not delivered for more than 15 minutes (ovhost alerts relay: Prometheus, the token or Network)"nnew file mode 100644@@@ -0,0 +1,19 @@++[Unit]++Description=OpenVibe operator alerts: the alerts firing in Prometheus, delivered to the operator through OpenVibe.Network++Documentation=https://github.com/OpenVibers/OpenVibe.Host/blob/main/docs/operator-alerts.md++After=network-online.target prometheus.service++Wants=network-online.target++++[Service]++Type=oneshot++# Root: /etc/openvibe/host.env (Host's service principal) is root-only and read by ovhost itself, and the++# textfile collector's directory is root's.++User=root++ExecStart=/usr/local/bin/ovhost alerts relay++Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin++Environment=OVHOST_INVENTORY=/etc/openvibe/host.json++UMask=0077++TimeoutStartSec=90s++PrivateTmp=true++NoNewPrivileges=true++ProtectSystem=fullnnew file mode 100644@@@ -0,0 +1,10 @@++[Unit]++Description=Deliver the firing OpenVibe alerts to the operator every two minutes (WS-H task 11)++++[Timer]++OnBootSec=2min++OnUnitActiveSec=2min++AccuracySec=15s++++[Install]++WantedBy=timers.targetnnew file mode 100644@@@ -0,0 +1,52 @@++# Operator alerts++++Roadmap WS-H task 11. The host's Prometheus evaluates the rules in++[deploy/prometheus/openvibe-rules.yml](../deploy/prometheus/openvibe-rules.yml): targets down, error++rates, disk and memory, backups, deploy drift, the developer path, the Tools job proof, the browser check++of each release and the relay itself. Until 2026-09-26 nothing delivered them.++++## The path++++```++Prometheus (127.0.0.1:9090) ──GET /api/v1/alerts──▶ ovhost alerts relay (openvibe-alerts.timer, every 2 min)++ │ Host's service token (network.operator.alert)++ ▼++ OpenVibe.Network POST /internal/operator/alerts++ (network.operator-alerts-request@1: the complete firing set)++ │++ ▼++ an OPERATOR_ALERT notification (category admin) to the owner account:++ the bell, web push and the realtime badge++```++++- **Opened**: an alert that was not firing pages at once. Severity `page` is a critical notification and++ `ticket` a high one.++- **Reminded**: the alert pages again once a day while it stays open.++- **Resolved**: a notice goes out when the alert drops out of the firing set.++- **Recipients**: `OWNER_USERNAME` in Network's environment, plus the comma-separated `OPERATOR_ALERT_USERNAMES`.++- **What leaves the host**: only the alert name, severity, summary, description, `service` label and start time.++ Instance and job labels stay on the host; the fingerprint is a hash of the labels.++- **Pending alerts** (inside their `for:` window) are not sent.++- **Prometheus unreachable**: the relay sends nothing, because an empty set would resolve every open alert.++- **Relay failures**: `openvibe_alert_relay_last_run_ok` and `openvibe_alert_relay_last_success_timestamp_seconds`++ record them, and `OpenVibeAlertRelayFailing` pages as soon as delivery works again.++++## Commands++++```bash++sudo ovhost alerts relay --dry-run # what is firing, nothing sent++sudo ovhost alerts relay # deliver now++systemctl list-timers openvibe-alerts.timer++```++++## Install (once)++++```bash++sudo cp /usr/local/lib/openvibe-host/deploy/systemd/openvibe-alerts.{service,timer} /etc/systemd/system/++sudo systemctl daemon-reload && sudo systemctl enable --now openvibe-alerts.timer++sudo cp /usr/local/lib/openvibe-host/deploy/prometheus/openvibe-rules.yml /etc/prometheus/rules/openvibe.yml++sudo promtool check rules /etc/prometheus/rules/openvibe.yml && sudo systemctl reload prometheus++```++++Network grants the `host` principal `network.operator.alert` (Network `server/identity/principals.js`), and the++relay uses the same credentials as release notifications, from `/etc/openvibe/host.env`.nnew file mode 100644@@@ -0,0 +1,134 @@++'use strict';++/**++ * `ovhost alerts relay` (roadmap WS-H task 11): the alerts firing in the host's Prometheus, delivered to the++ * operator. Prometheus evaluates the rules in deploy/prometheus/openvibe-rules.yml (backups, the developer path,++ * the Tools job proof, the browser check), but nothing delivered them. Every two minutes++ * (openvibe-alerts.timer) this reads GET <prometheus>/api/v1/alerts, keeps the firing ones (not pending), and++ * sends the complete set to OpenVibe.Network's POST /internal/operator/alerts (network.operator.alert,++ * network.operator-alerts-request@1) with Host's service token. Network pages the owner when an alert opens,++ * once a day while it stays open, and when it resolves (it drops out of the set).++ *++ * Per alert: fingerprint = sha256 of the alert name and its sorted labels (first 32 hex), name = alertname,++ * severity = the severity label (page or critical → critical, ticket or warning → warning, info; anything else is++ * warning), summary and description++ * = the rule's annotations, service = the service label when it is a service id, started_at = activeAt.++ * Free-form labels never leave the host.++ *++ * A Prometheus that does not answer sends nothing: an empty set would resolve every open alert. Each run++ * writes openvibe_alert_relay.prom to the textfile collector (last run, last success, alerts firing).++ * Credentials: Host's principal from /etc/openvibe/host.env (announce.js loadCredentials; the secret and the++ * token are never printed).++ */++const crypto = require('crypto');++const path = require('path');++const { loadCredentials } = require('./announce');++++const DEFAULT_PROMETHEUS = 'http://127.0.0.1:9090';++// The rules' severity label: page (and critical) pages now; ticket (and warning) pages at high; info is quiet.++const SEVERITIES = { critical: 'critical', page: 'critical', warning: 'warning', ticket: 'warning', info: 'info' };++const NAME_RE = /^[A-Za-z][A-Za-z0-9_:]{0,99}$/;++const SERVICE_RE = /^[a-z][a-z0-9-]{1,39}$/;++const TEXTFILE_DIR = '/var/lib/prometheus/node-exporter';++const MAX = 200;++++class AlertsError extends Error {++ constructor(message, exitCode = 2) { super(message); this.name = 'AlertsError'; this.exitCode = exitCode; }++}++++function fingerprint(labels) {++ const keys = Object.keys(labels || {}).sort();++ return crypto.createHash('sha256').update(JSON.stringify(keys.map((k) => [k, String(labels[k])]))).digest('hex').slice(0, 32);++}++++const clip = (s, n) => { const t = String(s || '').replace(/\s+/g, ' ').trim(); return t.length > n ? `${t.slice(0, n - 1)}…` : t; };++++/** Prometheus alerts (the /api/v1/alerts data) → network.operator-alerts-request@1 alerts. */++function toReport(promAlerts) {++ const out = new Map();++ for (const a of promAlerts || []) {++ if (!a || a.state !== 'firing') continue;++ const labels = a.labels || {};++ const name = String(labels.alertname || '');++ if (!NAME_RE.test(name)) continue;++ const ann = a.annotations || {};++ const started = Date.parse(a.activeAt);++ const item = {++ fingerprint: fingerprint(labels),++ name,++ severity: SEVERITIES[labels.severity] || 'warning',++ summary: clip(ann.summary || name, 300) || name,++ started_at: new Date(Number.isFinite(started) ? started : Date.now()).toISOString(),++ };++ const description = clip(ann.description, 1000);++ if (description) item.description = description;++ if (SERVICE_RE.test(String(labels.service || ''))) item.service = labels.service;++ out.set(item.fingerprint, item);++ }++ return [...out.values()].slice(0, MAX);++}++++async function writeMetrics(exec, dir, r) {++ if (!(await exec.readdir(dir))) return false;++ const now = Math.floor(Date.now() / 1000);++ let lastOk = r.ok ? now : null;++ if (!lastOk) {++ const prev = (await exec.readFile(path.join(dir, 'openvibe_alert_relay.prom'))) || '';++ const m = prev.match(/^openvibe_alert_relay_last_success_timestamp_seconds (\d+)$/m);++ lastOk = m ? Number(m[1]) : null;++ }++ const lines = [++ '# HELP openvibe_alert_relay_last_run_timestamp_seconds When ovhost alerts relay last ran.',++ '# TYPE openvibe_alert_relay_last_run_timestamp_seconds gauge',++ `openvibe_alert_relay_last_run_timestamp_seconds ${now}`,++ '# HELP openvibe_alert_relay_last_run_ok 1 when the last run delivered the firing set to Network.',++ '# TYPE openvibe_alert_relay_last_run_ok gauge',++ `openvibe_alert_relay_last_run_ok ${r.ok ? 1 : 0}`,++ '# HELP openvibe_alert_relay_firing Alerts firing at the last run that read Prometheus.',++ '# TYPE openvibe_alert_relay_firing gauge',++ `openvibe_alert_relay_firing ${r.firing == null ? 'NaN' : r.firing}`,++ ];++ if (lastOk) lines.push('# HELP openvibe_alert_relay_last_success_timestamp_seconds When a run last delivered.', '# TYPE openvibe_alert_relay_last_success_timestamp_seconds gauge', `openvibe_alert_relay_last_success_timestamp_seconds ${lastOk}`);++ await exec.writeFile(path.join(dir, 'openvibe_alert_relay.prom'), `${lines.join('\n')}\n`, { mode: 0o644 });++ return true;++}++++/**++ * One relay run. Answers { ok, firing, alerts, result?, stage?, error?, dryRun? }.++ * opts: prometheus (URL), envFile, env, dryRun (read and map only), metrics (default true).++ */++async function relay(ctx, { prometheus = null, envFile = null, env = {}, dryRun = false, metrics = true } = {}) {++ const { exec, inv } = ctx;++ const prom = String(prometheus || env.OVHOST_PROMETHEUS_URL || (inv.prometheus && inv.prometheus.url) || DEFAULT_PROMETHEUS).replace(/\/+$/, '');++ const done = async (r) => {++ if (metrics && !dryRun) { try { await writeMetrics(exec, inv.textfileDir || TEXTFILE_DIR, r); } catch { /* metrics never fail a run */ } }++ return r;++ };++ const p = await exec.request(`${prom}/api/v1/alerts`, { timeoutMs: 5000 });++ let data = null;++ try { data = p.status === 200 ? JSON.parse(p.body) : null; } catch { data = null; }++ if (!data || data.status !== 'success' || !data.data || !Array.isArray(data.data.alerts)) {++ return done({ ok: false, stage: 'prometheus', firing: null, error: p.status ? `Prometheus answered ${p.status}` : `Prometheus unreachable (${p.error || 'no answer'})` });++ }++ const alerts = toReport(data.data.alerts);++ const body = { source: 'prometheus', sent_at: new Date().toISOString(), alerts };++ if (dryRun) return { ok: true, dryRun: true, firing: alerts.length, alerts };++++ const creds = await loadCredentials(exec, inv, { file: envFile, env });++ if (creds.error) return done({ ok: false, stage: 'credentials', firing: alerts.length, alerts, error: creds.error });++ const form = new URLSearchParams({ grant_type: 'client_credentials', client_id: creds.clientId, client_secret: creds.secret, audience: 'openvibe.network', scope: 'network.operator.alert' });++ const t = await exec.request(`${creds.networkUrl}/oauth/token`, { method: 'POST', headers: { 'Content-Type': 'application/x-www-form-urlencoded', Accept: 'application/json' }, body: form.toString(), timeoutMs: 5000 });++ let token = null;++ try { token = t.status === 200 ? JSON.parse(t.body).access_token : null; } catch { token = null; }++ if (typeof token !== 'string' || !token) return done({ ok: false, stage: 'token', firing: alerts.length, alerts, error: t.status ? `the token endpoint answered ${t.status}` : `Network unreachable (${t.error || 'no answer'})` });++ const r = await exec.request(`${creds.networkUrl}/internal/operator/alerts`, {++ method: 'POST', headers: { 'Content-Type': 'application/json', Accept: 'application/json', Authorization: `Bearer ${token}` }, body: JSON.stringify(body), timeoutMs: 10000,++ });++ let result = null;++ try { result = JSON.parse(r.body); } catch { result = null; }++ if (r.status !== 200 || !result || result.ok !== true) {++ return done({ ok: false, stage: 'deliver', firing: alerts.length, alerts, error: r.status ? `Network answered ${r.status}${result && result.error ? `: ${result.error}` : ''}` : `Network unreachable (${r.error || 'no answer'})` });++ }++ return done({ ok: true, firing: alerts.length, alerts, result });++}++++module.exports = { relay, toReport, fingerprint, writeMetrics, AlertsError, DEFAULT_PROMETHEUS };@@@ -62,6 +62,9 @@ Usage: ovhost <command> [args] [--inventory <file>] [--json]declared by the inventory) to <prefix>-archive/; never deletes themarchive list the archives off-hostarchive restore <stamp> <path> --out <dir> one archived file back, checked against the manifest++ alerts relay [--prometheus <url>] [--dry-run]++ the alerts firing in Prometheus, sent to Network, which pages++ the operator (openvibe-alerts.timer, every 2 minutes)restore-download <service> <run|latest> [--out <dir>] [--from-host <name>]download, verify and decrypt one service's databases into aNEW root-only directory; never touches a live database@@@ -80,7 +83,7 @@ drill: 0 passed · 1 precondition (not root, port in use, no backup, unsupportedbackup --all / offsite: 0 everything succeeded · 1 precondition (config, key, lock) · 2 something failed`;--const FLAGS_WITH_VALUE = new Set(['inventory', 'to', 'limit', 'warn-days', 'variant', 'manifest', 'out', 'ready-timeout', 'wildcard-cert', 'backup', 'run', 'keep-daily', 'keep-weekly', 'from-host', 'backup-env', 'release', 'commit', 'origin', 'events-env', 'note']);++const FLAGS_WITH_VALUE = new Set(['prometheus', 'inventory', 'to', 'limit', 'warn-days', 'variant', 'manifest', 'out', 'ready-timeout', 'wildcard-cert', 'backup', 'run', 'keep-daily', 'keep-weekly', 'from-host', 'backup-env', 'release', 'commit', 'origin', 'events-env', 'note']);function parseArgs(argv) {const pos = [];@@@ -367,6 +370,16 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => procelse out(`OFFSITE ${o.ok ? 'ok' : 'FAILED'}: ${o.uploaded.length} file(s) to s3://${o.bucket}/${o.location}${o.failures.length ? `; failures: ${o.failures.map((f) => `${f.service ? `${f.service}/` : ''}${f.file}: ${f.error}`).join('; ')}` : ''}`);return o.ok ? 0 : 2;}++ case 'alerts': {++ // WS-H task 11: deliver the firing Prometheus alerts to the operator through Network.++ if (pos[1] !== 'relay') throw new UsageError('usage: ovhost alerts relay [--prometheus <url>] [--dry-run] [--events-env <file>]');++ const r = await lib.alerts.relay(ctx, { prometheus: opts.prometheus, envFile: opts['events-env'], env, dryRun: !!opts['dry-run'] });++ if (json) emit(r);++ else if (!r.ok) out(`ALERTS not delivered (${r.stage}): ${r.error}`);++ else if (r.dryRun) { out(`${r.firing} alert(s) firing (dry run, nothing sent)`); for (const a of r.alerts) out(` ${a.severity.padEnd(8)} ${a.name} ${a.summary}`); }++ else out(`ALERTS delivered: ${r.firing} firing; opened ${r.result.opened}, reminded ${r.result.reminded}, resolved ${r.result.resolved}; ${r.result.notified} notification(s)`);++ return r.ok ? 0 : 2;++ }case 'archive': {// WS-S task 6: dead and backup database files, encrypted off-host before the owner deletes them.const sub = pos[1];@@@ -34,6 +34,7 @@ module.exports = {BackupError,offsite,archive: require('./archive'),++ alerts: require('./alerts'),snapshot,drill,DrillError,@@@ -16,7 +16,7 @@"dotenv": "^16.4.7","express": "^4.21.2","express-rate-limit": "^7.5.0",-- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.58.0",++ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.62.0","openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.11.0","openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.22.0"},@@@ -1412,9 +1412,9 @@}},"node_modules/openvibe-contracts": {-- "version": "0.58.0",-- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.58.0",-- "integrity": "sha512-pIKhO6XgbCzGkBN3JVXDoDAaGyAHxdX3hlmSpKetszCguBDeK4TDre/44j2+eiqzRadM7OdJ655GKGOvvtKVhQ==",++ "version": "0.62.0",++ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.62.0",++ "integrity": "sha512-mmVQToHBY77NrZEHqOBoPuRdZ/bnAzp11E7VkBrN9maPdGCoG8YIJqIq/Hi/dFiDrLJtAcdXRKl/tahdWrkkdQ==","license": "AGPL-3.0-only","dependencies": {"ajv": "^8.17.1",@@@ -31,7 +31,7 @@"dotenv": "^16.4.7","express": "^4.21.2","express-rate-limit": "^7.5.0",-- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.58.0",++ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.62.0","openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.11.0","openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.22.0"},nnew file mode 100644@@@ -0,0 +1,133 @@++'use strict';++/**++ * Operator alerts (lib/alerts.js, WS-H task 11): `ovhost alerts relay` reads the firing alerts from Prometheus++ * and sends the complete set to Network's POST /internal/operator/alerts with Host's token++ * (network.operator.alert); a Prometheus that does not answer sends nothing; secrets never show; each run++ * leaves openvibe_alert_relay.prom for the textfile collector.++ */++const assert = require('assert');++const contracts = require('openvibe-contracts');++const { scenario, test, runTests, SECRET } = require('./helpers');++const { toReport, fingerprint } = require('../lib/alerts');++++const TOKEN = 'tok_ALERTS_TOKEN_MUST_NEVER_APPEAR_5d1e';++const HOST_SECRET = `${SECRET}-host`;++const PROM_FILE = '/var/lib/prometheus/node-exporter/openvibe_alert_relay.prom';++++const FIRING = [++ { labels: { alertname: 'OpenVibeBackupMissed', severity: 'page', instance: '127.0.0.1:9100', job: 'node' }, annotations: { summary: 'No successful nightly backup for more than 30 hours' }, state: 'firing', activeAt: '2026-09-26T05:35:00.123Z', value: '1' },++ { labels: { alertname: 'OpenVibeDeployDrift', severity: 'ticket', service: 'live' }, annotations: { summary: 'live: main has had commits production does not run for more than a day', description: 'x'.repeat(1500) }, state: 'firing', activeAt: '2026-09-26T06:00:00Z' },++ { labels: { alertname: 'OpenVibeToolsJobProofFailed', severity: 'page' }, annotations: {}, state: 'pending', activeAt: '2026-09-26T13:59:00Z' },++ { labels: { alertname: 'bad name!' }, state: 'firing', activeAt: '2026-09-26T06:00:00Z' },++];++++function withNetwork({ prom = { status: 200, body: { status: 'success', data: { alerts: FIRING } } }, network = 'ok' } = {}) {++ const host = scenario();++ host.put('/etc/openvibe/host.env', `OV_OAUTH_CLIENT_ID=host\nOV_OAUTH_CLIENT_SECRET="${HOST_SECRET}"\nEVENTS_URL=http://127.0.0.1:4300\nOV_NETWORK_INTERNAL_URL=http://127.0.0.1:4000\n`, { mode: 0o600, owner: 'root' });++ host.put('/var/lib/prometheus/node-exporter/.keep', '');++ host.tokenRequests = [];++ host.delivered = [];++ if (prom) host.http.set('http://127.0.0.1:9090/api/v1/alerts', () => prom);++ host.http.set('http://127.0.0.1:4000/oauth/token', ({ body }) => {++ const form = Object.fromEntries(new URLSearchParams(body));++ host.tokenRequests.push(form);++ if (form.client_secret !== HOST_SECRET) return { status: 401, body: { error: 'invalid_client' } };++ return { status: 200, body: { access_token: TOKEN, token_type: 'Bearer', expires_in: 300 } };++ });++ host.http.set('http://127.0.0.1:4000/internal/operator/alerts', ({ method, headers, body }) => {++ assert.strictEqual(method, 'POST');++ assert.strictEqual(headers.Authorization, `Bearer ${TOKEN}`);++ const doc = JSON.parse(body);++ host.delivered.push(doc);++ if (network !== 'ok') return network;++ return { status: 200, body: { ok: true, firing: doc.alerts.length, opened: doc.alerts.length, reminded: 0, resolved: 0, notified: doc.alerts.length } };++ });++ return host;++}++++function noSecrets(host, out) {++ for (const s of [HOST_SECRET, TOKEN]) {++ assert.ok(!out.includes(s), 'the output shows no secret');++ for (const [p, f] of host.files) if (p !== '/etc/openvibe/host.env' && f.type === 'file') assert.ok(!String(f.content).includes(s), `${p} holds no secret`);++ }++}++++runTests([++ test('toReport: firing only, the rule severities mapped, annotations clipped, no free-form labels, stable fingerprints', async () => {++ const r = toReport(FIRING);++ assert.deepStrictEqual(r.map((a) => [a.name, a.severity, a.service || null]), [['OpenVibeBackupMissed', 'critical', null], ['OpenVibeDeployDrift', 'warning', 'live']]);++ assert.strictEqual(r[0].started_at, '2026-09-26T05:35:00.123Z');++ assert.strictEqual(r[1].description.length, 1000);++ assert.ok(!JSON.stringify(r).includes('127.0.0.1:9100'), 'instance and job labels stay on the host');++ assert.strictEqual(r[0].fingerprint, fingerprint({ severity: 'page', alertname: 'OpenVibeBackupMissed', job: 'node', instance: '127.0.0.1:9100' }), 'label order does not matter');++ assert.notStrictEqual(r[0].fingerprint, r[1].fingerprint);++ const v = contracts.validate('network.operator-alerts-request@1', { source: 'prometheus', alerts: r });++ assert.ok(v.valid, JSON.stringify(v.errors));++ assert.deepStrictEqual(toReport([]), []);++ }),++ test('alerts relay: a token for network.operator.alert, the complete firing set delivered, metrics written', async () => {++ const host = withNetwork();++ const r = await host.cli('alerts', 'relay');++ assert.strictEqual(r.code, 0, r.out);++ assert.match(r.out, /ALERTS delivered: 2 firing; opened 2, reminded 0, resolved 0; 2 notification\(s\)/);++ assert.deepStrictEqual(host.tokenRequests.map((t) => [t.grant_type, t.client_id, t.audience, t.scope]), [['client_credentials', 'host', 'openvibe.network', 'network.operator.alert']]);++ assert.strictEqual(host.delivered.length, 1);++ const doc = host.delivered[0];++ assert.ok(contracts.validate('network.operator-alerts-request@1', doc).valid);++ assert.deepStrictEqual(doc.alerts.map((a) => a.name), ['OpenVibeBackupMissed', 'OpenVibeDeployDrift']);++ const prom = host.files.get(PROM_FILE).content;++ assert.match(prom, /^openvibe_alert_relay_last_run_ok 1$/m);++ assert.match(prom, /^openvibe_alert_relay_firing 2$/m);++ assert.match(prom, /^openvibe_alert_relay_last_success_timestamp_seconds \d+$/m);++ noSecrets(host, r.out);++ }),++ test('alerts relay: nothing firing sends an empty set (Network resolves what was open)', async () => {++ const host = withNetwork({ prom: { status: 200, body: { status: 'success', data: { alerts: [] } } } });++ const r = await host.cli('alerts', 'relay');++ assert.strictEqual(r.code, 0, r.out);++ assert.deepStrictEqual(host.delivered.map((d) => d.alerts), [[]]);++ }),++ test('alerts relay: Prometheus down or odd → nothing is sent (an empty set would resolve everything), exit 2', async () => {++ for (const prom of [null, { status: 503, body: 'no' }, { status: 200, body: { status: 'error' } }]) {++ const host = withNetwork({ prom });++ const r = await host.cli('alerts', 'relay');++ assert.strictEqual(r.code, 2, r.out);++ assert.match(r.out, /ALERTS not delivered \(prometheus\)/);++ assert.strictEqual(host.delivered.length, 0);++ assert.strictEqual(host.tokenRequests.length, 0);++ assert.match(host.files.get(PROM_FILE).content, /^openvibe_alert_relay_last_run_ok 0$/m);++ assert.match(host.files.get(PROM_FILE).content, /^openvibe_alert_relay_firing NaN$/m);++ }++ }),++ test('alerts relay: Network refusing → exit 2 with the stage; the last success is kept from the previous file', async () => {++ const host = withNetwork({ network: { status: 403, body: { error: 'Forbidden' } } });++ host.put(PROM_FILE, 'openvibe_alert_relay_last_success_timestamp_seconds 1790000000\n');++ const r = await host.cli('alerts', 'relay');++ assert.strictEqual(r.code, 2);++ assert.match(r.out, /ALERTS not delivered \(deliver\): Network answered 403: Forbidden/);++ assert.match(host.files.get(PROM_FILE).content, /^openvibe_alert_relay_last_success_timestamp_seconds 1790000000$/m);++ noSecrets(host, r.out);++ }),++ test('alerts relay --dry-run: reads and maps, asks for no token, sends nothing; --json', async () => {++ const host = withNetwork();++ const r = await host.cli('alerts', 'relay', '--dry-run');++ assert.strictEqual(r.code, 0);++ assert.match(r.out, /2 alert\(s\) firing \(dry run, nothing sent\)/);++ assert.match(r.out, /critical +OpenVibeBackupMissed/);++ assert.deepStrictEqual([host.tokenRequests.length, host.delivered.length], [0, 0]);++ const j = await withNetwork().cli('alerts', 'relay', '--json');++ assert.strictEqual(JSON.parse(j.out).result.opened, 2);++ }),++ test('alerts relay: no credentials → exit 2 at the credentials stage', async () => {++ const host = withNetwork();++ host.files.delete('/etc/openvibe/host.env');++ const r = await host.cli('alerts', 'relay');++ assert.strictEqual(r.code, 2);++ assert.match(r.out, /ALERTS not delivered \(credentials\)/);++ }),++ test('alerts: usage', async () => {++ const r = await scenario().cli('alerts');++ assert.notStrictEqual(r.code, 0);++ assert.match(r.out, /usage: ovhost alerts relay/);++ }),++]);
@@ -1,6 +1,8 @@
@@ -1,6 +1,8 @@
# OpenVibe alert rules (roadmap Track O), loaded by the host's Prometheus (rule_files in the config
# OpenVibe alert rules (roadmap Track O), loaded by the host's Prometheus (rule_files in the config
-# scripts/prometheus-config.js renders). Firing alerts are read at GET 127.0.0.1:9090/api/v1/alerts;
-# no notifier is configured yet.
+# scripts/prometheus-config.js renders). Firing alerts are read at GET 127.0.0.1:9090/api/v1/alerts by
+# `ovhost alerts relay` (openvibe-alerts.timer, every 2 minutes; lib/alerts.js), which sends them to
+# OpenVibe.Network: severity page pages the operator at once (critical), ticket at high priority, and each
+# alert is reminded daily while open and announced when it resolves (docs/operator-alerts.md).
groups:
groups:
- name: openvibe
- name: openvibe
rules:
rules:
@@ -77,3 +79,11 @@ groups:
@@ -77,3 +79,11 @@ groups:
labels: { severity: ticket }
labels: { severity: ticket }
annotations:
annotations:
summary: "No passing Tools job proof for more than 13 hours (two runs)"
summary: "No passing Tools job proof for more than 13 hours (two runs)"
+ # The relay itself (openvibe-alerts.timer; WS-H task 11). It cannot page while it is broken; it pages
+ # this as soon as a run gets through again, so an outage of the pager is never silent afterwards.
+ - alert: OpenVibeAlertRelayFailing
+ expr: time() - openvibe_alert_relay_last_success_timestamp_seconds > 15 * 60
+ for: 5m
+ labels: { severity: page }
+ annotations:
+ summary: "Operator alerts were not delivered for more than 15 minutes (ovhost alerts relay: Prometheus, the token or Network)"
new file mode 100644
new file mode 100644
@@ -0,0 +1,19 @@
@@ -0,0 +1,19 @@
+[Unit]
+Description=OpenVibe operator alerts: the alerts firing in Prometheus, delivered to the operator through OpenVibe.Network
+Documentation=https://github.com/OpenVibers/OpenVibe.Host/blob/main/docs/operator-alerts.md
+After=network-online.target prometheus.service
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+# Root: /etc/openvibe/host.env (Host's service principal) is root-only and read by ovhost itself, and the
+# textfile collector's directory is root's.
+User=root
+ExecStart=/usr/local/bin/ovhost alerts relay
+Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+Environment=OVHOST_INVENTORY=/etc/openvibe/host.json
+UMask=0077
+TimeoutStartSec=90s
+PrivateTmp=true
+NoNewPrivileges=true
+ProtectSystem=full
new file mode 100644
new file mode 100644
@@ -0,0 +1,10 @@
@@ -0,0 +1,10 @@
+[Unit]
+Description=Deliver the firing OpenVibe alerts to the operator every two minutes (WS-H task 11)
+
+[Timer]
+OnBootSec=2min
+OnUnitActiveSec=2min
+AccuracySec=15s
+
+[Install]
+WantedBy=timers.target
new file mode 100644
new file mode 100644
@@ -0,0 +1,52 @@
@@ -0,0 +1,52 @@
+# Operator alerts
+
+Roadmap WS-H task 11. The host's Prometheus evaluates the rules in
+[deploy/prometheus/openvibe-rules.yml](../deploy/prometheus/openvibe-rules.yml): targets down, error
+rates, disk and memory, backups, deploy drift, the developer path, the Tools job proof, the browser check
+of each release and the relay itself. Until 2026-09-26 nothing delivered them.
+
+## The path
+
+```
+Prometheus (127.0.0.1:9090) ──GET /api/v1/alerts──▶ ovhost alerts relay (openvibe-alerts.timer, every 2 min)
+ │ Host's service token (network.operator.alert)
+ ▼
+ OpenVibe.Network POST /internal/operator/alerts
+ (network.operator-alerts-request@1: the complete firing set)
+ │
+ ▼
+ an OPERATOR_ALERT notification (category admin) to the owner account:
+ the bell, web push and the realtime badge
+```
+
+- **Opened**: an alert that was not firing pages at once. Severity `page` is a critical notification and
+ `ticket` a high one.
+- **Reminded**: the alert pages again once a day while it stays open.
+- **Resolved**: a notice goes out when the alert drops out of the firing set.
+- **Recipients**: `OWNER_USERNAME` in Network's environment, plus the comma-separated `OPERATOR_ALERT_USERNAMES`.
+- **What leaves the host**: only the alert name, severity, summary, description, `service` label and start time.
+ Instance and job labels stay on the host; the fingerprint is a hash of the labels.
+- **Pending alerts** (inside their `for:` window) are not sent.
+- **Prometheus unreachable**: the relay sends nothing, because an empty set would resolve every open alert.
+- **Relay failures**: `openvibe_alert_relay_last_run_ok` and `openvibe_alert_relay_last_success_timestamp_seconds`
+ record them, and `OpenVibeAlertRelayFailing` pages as soon as delivery works again.
+
+## Commands
+
+```bash
+sudo ovhost alerts relay --dry-run # what is firing, nothing sent
+sudo ovhost alerts relay # deliver now
+systemctl list-timers openvibe-alerts.timer
+```
+
+## Install (once)
+
+```bash
+sudo cp /usr/local/lib/openvibe-host/deploy/systemd/openvibe-alerts.{service,timer} /etc/systemd/system/
+sudo systemctl daemon-reload && sudo systemctl enable --now openvibe-alerts.timer
+sudo cp /usr/local/lib/openvibe-host/deploy/prometheus/openvibe-rules.yml /etc/prometheus/rules/openvibe.yml
+sudo promtool check rules /etc/prometheus/rules/openvibe.yml && sudo systemctl reload prometheus
+```
+
+Network grants the `host` principal `network.operator.alert` (Network `server/identity/principals.js`), and the
+relay uses the same credentials as release notifications, from `/etc/openvibe/host.env`.
new file mode 100644
new file mode 100644
@@ -0,0 +1,134 @@
@@ -0,0 +1,134 @@
+'use strict';
+/**
+ * `ovhost alerts relay` (roadmap WS-H task 11): the alerts firing in the host's Prometheus, delivered to the
+ * operator. Prometheus evaluates the rules in deploy/prometheus/openvibe-rules.yml (backups, the developer path,
+ * the Tools job proof, the browser check), but nothing delivered them. Every two minutes
+ * (openvibe-alerts.timer) this reads GET <prometheus>/api/v1/alerts, keeps the firing ones (not pending), and
+ * sends the complete set to OpenVibe.Network's POST /internal/operator/alerts (network.operator.alert,
+ * network.operator-alerts-request@1) with Host's service token. Network pages the owner when an alert opens,
+ * once a day while it stays open, and when it resolves (it drops out of the set).
+ *
+ * Per alert: fingerprint = sha256 of the alert name and its sorted labels (first 32 hex), name = alertname,
+ * severity = the severity label (page or critical → critical, ticket or warning → warning, info; anything else is
+ * warning), summary and description
+ * = the rule's annotations, service = the service label when it is a service id, started_at = activeAt.
+ * Free-form labels never leave the host.
+ *
+ * A Prometheus that does not answer sends nothing: an empty set would resolve every open alert. Each run
+ * writes openvibe_alert_relay.prom to the textfile collector (last run, last success, alerts firing).
+ * Credentials: Host's principal from /etc/openvibe/host.env (announce.js loadCredentials; the secret and the
+ * token are never printed).
+ */
+const crypto = require('crypto');
+const path = require('path');
+const { loadCredentials } = require('./announce');
+
+const DEFAULT_PROMETHEUS = 'http://127.0.0.1:9090';
+// The rules' severity label: page (and critical) pages now; ticket (and warning) pages at high; info is quiet.
+const SEVERITIES = { critical: 'critical', page: 'critical', warning: 'warning', ticket: 'warning', info: 'info' };
+const NAME_RE = /^[A-Za-z][A-Za-z0-9_:]{0,99}$/;
+const SERVICE_RE = /^[a-z][a-z0-9-]{1,39}$/;
+const TEXTFILE_DIR = '/var/lib/prometheus/node-exporter';
+const MAX = 200;
+
+class AlertsError extends Error {
+ constructor(message, exitCode = 2) { super(message); this.name = 'AlertsError'; this.exitCode = exitCode; }
+}
+
+function fingerprint(labels) {
+ const keys = Object.keys(labels || {}).sort();
+ return crypto.createHash('sha256').update(JSON.stringify(keys.map((k) => [k, String(labels[k])]))).digest('hex').slice(0, 32);
+}
+
+const clip = (s, n) => { const t = String(s || '').replace(/\s+/g, ' ').trim(); return t.length > n ? `${t.slice(0, n - 1)}…` : t; };
+
+/** Prometheus alerts (the /api/v1/alerts data) → network.operator-alerts-request@1 alerts. */
+function toReport(promAlerts) {
+ const out = new Map();
+ for (const a of promAlerts || []) {
+ if (!a || a.state !== 'firing') continue;
+ const labels = a.labels || {};
+ const name = String(labels.alertname || '');
+ if (!NAME_RE.test(name)) continue;
+ const ann = a.annotations || {};
+ const started = Date.parse(a.activeAt);
+ const item = {
+ fingerprint: fingerprint(labels),
+ name,
+ severity: SEVERITIES[labels.severity] || 'warning',
+ summary: clip(ann.summary || name, 300) || name,
+ started_at: new Date(Number.isFinite(started) ? started : Date.now()).toISOString(),
+ };
+ const description = clip(ann.description, 1000);
+ if (description) item.description = description;
+ if (SERVICE_RE.test(String(labels.service || ''))) item.service = labels.service;
+ out.set(item.fingerprint, item);
+ }
+ return [...out.values()].slice(0, MAX);
+}
+
+async function writeMetrics(exec, dir, r) {
+ if (!(await exec.readdir(dir))) return false;
+ const now = Math.floor(Date.now() / 1000);
+ let lastOk = r.ok ? now : null;
+ if (!lastOk) {
+ const prev = (await exec.readFile(path.join(dir, 'openvibe_alert_relay.prom'))) || '';
+ const m = prev.match(/^openvibe_alert_relay_last_success_timestamp_seconds (\d+)$/m);
+ lastOk = m ? Number(m[1]) : null;
+ }
+ const lines = [
+ '# HELP openvibe_alert_relay_last_run_timestamp_seconds When ovhost alerts relay last ran.',
+ '# TYPE openvibe_alert_relay_last_run_timestamp_seconds gauge',
+ `openvibe_alert_relay_last_run_timestamp_seconds ${now}`,
+ '# HELP openvibe_alert_relay_last_run_ok 1 when the last run delivered the firing set to Network.',
+ '# TYPE openvibe_alert_relay_last_run_ok gauge',
+ `openvibe_alert_relay_last_run_ok ${r.ok ? 1 : 0}`,
+ '# HELP openvibe_alert_relay_firing Alerts firing at the last run that read Prometheus.',
+ '# TYPE openvibe_alert_relay_firing gauge',
+ `openvibe_alert_relay_firing ${r.firing == null ? 'NaN' : r.firing}`,
+ ];
+ if (lastOk) lines.push('# HELP openvibe_alert_relay_last_success_timestamp_seconds When a run last delivered.', '# TYPE openvibe_alert_relay_last_success_timestamp_seconds gauge', `openvibe_alert_relay_last_success_timestamp_seconds ${lastOk}`);
+ await exec.writeFile(path.join(dir, 'openvibe_alert_relay.prom'), `${lines.join('\n')}\n`, { mode: 0o644 });
+ return true;
+}
+
+/**
+ * One relay run. Answers { ok, firing, alerts, result?, stage?, error?, dryRun? }.
+ * opts: prometheus (URL), envFile, env, dryRun (read and map only), metrics (default true).
+ */
+async function relay(ctx, { prometheus = null, envFile = null, env = {}, dryRun = false, metrics = true } = {}) {
+ const { exec, inv } = ctx;
+ const prom = String(prometheus || env.OVHOST_PROMETHEUS_URL || (inv.prometheus && inv.prometheus.url) || DEFAULT_PROMETHEUS).replace(/\/+$/, '');
+ const done = async (r) => {
+ if (metrics && !dryRun) { try { await writeMetrics(exec, inv.textfileDir || TEXTFILE_DIR, r); } catch { /* metrics never fail a run */ } }
+ return r;
+ };
+ const p = await exec.request(`${prom}/api/v1/alerts`, { timeoutMs: 5000 });
+ let data = null;
+ try { data = p.status === 200 ? JSON.parse(p.body) : null; } catch { data = null; }
+ if (!data || data.status !== 'success' || !data.data || !Array.isArray(data.data.alerts)) {
+ return done({ ok: false, stage: 'prometheus', firing: null, error: p.status ? `Prometheus answered ${p.status}` : `Prometheus unreachable (${p.error || 'no answer'})` });
+ }
+ const alerts = toReport(data.data.alerts);
+ const body = { source: 'prometheus', sent_at: new Date().toISOString(), alerts };
+ if (dryRun) return { ok: true, dryRun: true, firing: alerts.length, alerts };
+
+ const creds = await loadCredentials(exec, inv, { file: envFile, env });
+ if (creds.error) return done({ ok: false, stage: 'credentials', firing: alerts.length, alerts, error: creds.error });
+ const form = new URLSearchParams({ grant_type: 'client_credentials', client_id: creds.clientId, client_secret: creds.secret, audience: 'openvibe.network', scope: 'network.operator.alert' });
+ const t = await exec.request(`${creds.networkUrl}/oauth/token`, { method: 'POST', headers: { 'Content-Type': 'application/x-www-form-urlencoded', Accept: 'application/json' }, body: form.toString(), timeoutMs: 5000 });
+ let token = null;
+ try { token = t.status === 200 ? JSON.parse(t.body).access_token : null; } catch { token = null; }
+ if (typeof token !== 'string' || !token) return done({ ok: false, stage: 'token', firing: alerts.length, alerts, error: t.status ? `the token endpoint answered ${t.status}` : `Network unreachable (${t.error || 'no answer'})` });
+ const r = await exec.request(`${creds.networkUrl}/internal/operator/alerts`, {
+ method: 'POST', headers: { 'Content-Type': 'application/json', Accept: 'application/json', Authorization: `Bearer ${token}` }, body: JSON.stringify(body), timeoutMs: 10000,
+ });
+ let result = null;
+ try { result = JSON.parse(r.body); } catch { result = null; }
+ if (r.status !== 200 || !result || result.ok !== true) {
+ return done({ ok: false, stage: 'deliver', firing: alerts.length, alerts, error: r.status ? `Network answered ${r.status}${result && result.error ? `: ${result.error}` : ''}` : `Network unreachable (${r.error || 'no answer'})` });
+ }
+ return done({ ok: true, firing: alerts.length, alerts, result });
+}
+
+module.exports = { relay, toReport, fingerprint, writeMetrics, AlertsError, DEFAULT_PROMETHEUS };
@@ -62,6 +62,9 @@ Usage: ovhost <command> [args] [--inventory <file>] [--json]
@@ -62,6 +62,9 @@ Usage: ovhost <command> [args] [--inventory <file>] [--json]
declared by the inventory) to <prefix>-archive/; never deletes them
declared by the inventory) to <prefix>-archive/; never deletes them
archive list the archives off-host
archive list the archives off-host
archive restore <stamp> <path> --out <dir> one archived file back, checked against the manifest
archive restore <stamp> <path> --out <dir> one archived file back, checked against the manifest
+ alerts relay [--prometheus <url>] [--dry-run]
+ the alerts firing in Prometheus, sent to Network, which pages
+ the operator (openvibe-alerts.timer, every 2 minutes)
restore-download <service> <run|latest> [--out <dir>] [--from-host <name>]
restore-download <service> <run|latest> [--out <dir>] [--from-host <name>]
download, verify and decrypt one service's databases into a
download, verify and decrypt one service's databases into a
NEW root-only directory; never touches a live database
NEW root-only directory; never touches a live database
@@ -80,7 +83,7 @@ drill: 0 passed · 1 precondition (not root, port in use, no backup, unsupported
@@ -80,7 +83,7 @@ drill: 0 passed · 1 precondition (not root, port in use, no backup, unsupported
backup --all / offsite: 0 everything succeeded · 1 precondition (config, key, lock) · 2 something failed
backup --all / offsite: 0 everything succeeded · 1 precondition (config, key, lock) · 2 something failed
`;
`;
-const FLAGS_WITH_VALUE = new Set(['inventory', 'to', 'limit', 'warn-days', 'variant', 'manifest', 'out', 'ready-timeout', 'wildcard-cert', 'backup', 'run', 'keep-daily', 'keep-weekly', 'from-host', 'backup-env', 'release', 'commit', 'origin', 'events-env', 'note']);
+const FLAGS_WITH_VALUE = new Set(['prometheus', 'inventory', 'to', 'limit', 'warn-days', 'variant', 'manifest', 'out', 'ready-timeout', 'wildcard-cert', 'backup', 'run', 'keep-daily', 'keep-weekly', 'from-host', 'backup-env', 'release', 'commit', 'origin', 'events-env', 'note']);
function parseArgs(argv) {
function parseArgs(argv) {
const pos = [];
const pos = [];
@@ -367,6 +370,16 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc
@@ -367,6 +370,16 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc
else out(`OFFSITE ${o.ok ? 'ok' : 'FAILED'}: ${o.uploaded.length} file(s) to s3://${o.bucket}/${o.location}${o.failures.length ? `; failures: ${o.failures.map((f) => `${f.service ? `${f.service}/` : ''}${f.file}: ${f.error}`).join('; ')}` : ''}`);
else out(`OFFSITE ${o.ok ? 'ok' : 'FAILED'}: ${o.uploaded.length} file(s) to s3://${o.bucket}/${o.location}${o.failures.length ? `; failures: ${o.failures.map((f) => `${f.service ? `${f.service}/` : ''}${f.file}: ${f.error}`).join('; ')}` : ''}`);
return o.ok ? 0 : 2;
return o.ok ? 0 : 2;
}
}
+ case 'alerts': {
+ // WS-H task 11: deliver the firing Prometheus alerts to the operator through Network.
+ if (pos[1] !== 'relay') throw new UsageError('usage: ovhost alerts relay [--prometheus <url>] [--dry-run] [--events-env <file>]');
+ const r = await lib.alerts.relay(ctx, { prometheus: opts.prometheus, envFile: opts['events-env'], env, dryRun: !!opts['dry-run'] });
+ if (json) emit(r);
+ else if (!r.ok) out(`ALERTS not delivered (${r.stage}): ${r.error}`);
+ else if (r.dryRun) { out(`${r.firing} alert(s) firing (dry run, nothing sent)`); for (const a of r.alerts) out(` ${a.severity.padEnd(8)} ${a.name} ${a.summary}`); }
+ else out(`ALERTS delivered: ${r.firing} firing; opened ${r.result.opened}, reminded ${r.result.reminded}, resolved ${r.result.resolved}; ${r.result.notified} notification(s)`);
+ return r.ok ? 0 : 2;
+ }
case 'archive': {
case 'archive': {
// WS-S task 6: dead and backup database files, encrypted off-host before the owner deletes them.
// WS-S task 6: dead and backup database files, encrypted off-host before the owner deletes them.
const sub = pos[1];
const sub = pos[1];
@@ -34,6 +34,7 @@ module.exports = {
@@ -34,6 +34,7 @@ module.exports = {
BackupError,
BackupError,
offsite,
offsite,
archive: require('./archive'),
archive: require('./archive'),
+ alerts: require('./alerts'),
snapshot,
snapshot,
drill,
drill,
DrillError,
DrillError,
@@ -16,7 +16,7 @@
@@ -16,7 +16,7 @@
"dotenv": "^16.4.7",
"dotenv": "^16.4.7",
"express": "^4.21.2",
"express": "^4.21.2",
"express-rate-limit": "^7.5.0",
"express-rate-limit": "^7.5.0",
- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.58.0",
+ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.62.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.11.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.11.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.22.0"
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.22.0"
},
},
@@ -1412,9 +1412,9 @@
@@ -1412,9 +1412,9 @@
}
}
},
},
"node_modules/openvibe-contracts": {
"node_modules/openvibe-contracts": {
- "version": "0.58.0",
- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.58.0",
- "integrity": "sha512-pIKhO6XgbCzGkBN3JVXDoDAaGyAHxdX3hlmSpKetszCguBDeK4TDre/44j2+eiqzRadM7OdJ655GKGOvvtKVhQ==",
+ "version": "0.62.0",
+ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.62.0",
+ "integrity": "sha512-mmVQToHBY77NrZEHqOBoPuRdZ/bnAzp11E7VkBrN9maPdGCoG8YIJqIq/Hi/dFiDrLJtAcdXRKl/tahdWrkkdQ==",
"license": "AGPL-3.0-only",
"license": "AGPL-3.0-only",
"dependencies": {
"dependencies": {
"ajv": "^8.17.1",
"ajv": "^8.17.1",
@@ -31,7 +31,7 @@
@@ -31,7 +31,7 @@
"dotenv": "^16.4.7",
"dotenv": "^16.4.7",
"express": "^4.21.2",
"express": "^4.21.2",
"express-rate-limit": "^7.5.0",
"express-rate-limit": "^7.5.0",
- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.58.0",
+ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.62.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.11.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.11.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.22.0"
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.22.0"
},
},
new file mode 100644
new file mode 100644
@@ -0,0 +1,133 @@
@@ -0,0 +1,133 @@
+'use strict';
+/**
+ * Operator alerts (lib/alerts.js, WS-H task 11): `ovhost alerts relay` reads the firing alerts from Prometheus
+ * and sends the complete set to Network's POST /internal/operator/alerts with Host's token
+ * (network.operator.alert); a Prometheus that does not answer sends nothing; secrets never show; each run
+ * leaves openvibe_alert_relay.prom for the textfile collector.
+ */
+const assert = require('assert');
+const contracts = require('openvibe-contracts');
+const { scenario, test, runTests, SECRET } = require('./helpers');
+const { toReport, fingerprint } = require('../lib/alerts');
+
+const TOKEN = 'tok_ALERTS_TOKEN_MUST_NEVER_APPEAR_5d1e';
+const HOST_SECRET = `${SECRET}-host`;
+const PROM_FILE = '/var/lib/prometheus/node-exporter/openvibe_alert_relay.prom';
+
+const FIRING = [
+ { labels: { alertname: 'OpenVibeBackupMissed', severity: 'page', instance: '127.0.0.1:9100', job: 'node' }, annotations: { summary: 'No successful nightly backup for more than 30 hours' }, state: 'firing', activeAt: '2026-09-26T05:35:00.123Z', value: '1' },
+ { labels: { alertname: 'OpenVibeDeployDrift', severity: 'ticket', service: 'live' }, annotations: { summary: 'live: main has had commits production does not run for more than a day', description: 'x'.repeat(1500) }, state: 'firing', activeAt: '2026-09-26T06:00:00Z' },
+ { labels: { alertname: 'OpenVibeToolsJobProofFailed', severity: 'page' }, annotations: {}, state: 'pending', activeAt: '2026-09-26T13:59:00Z' },
+ { labels: { alertname: 'bad name!' }, state: 'firing', activeAt: '2026-09-26T06:00:00Z' },
+];
+
+function withNetwork({ prom = { status: 200, body: { status: 'success', data: { alerts: FIRING } } }, network = 'ok' } = {}) {
+ const host = scenario();
+ host.put('/etc/openvibe/host.env', `OV_OAUTH_CLIENT_ID=host\nOV_OAUTH_CLIENT_SECRET="${HOST_SECRET}"\nEVENTS_URL=http://127.0.0.1:4300\nOV_NETWORK_INTERNAL_URL=http://127.0.0.1:4000\n`, { mode: 0o600, owner: 'root' });
+ host.put('/var/lib/prometheus/node-exporter/.keep', '');
+ host.tokenRequests = [];
+ host.delivered = [];
+ if (prom) host.http.set('http://127.0.0.1:9090/api/v1/alerts', () => prom);
+ host.http.set('http://127.0.0.1:4000/oauth/token', ({ body }) => {
+ const form = Object.fromEntries(new URLSearchParams(body));
+ host.tokenRequests.push(form);
+ if (form.client_secret !== HOST_SECRET) return { status: 401, body: { error: 'invalid_client' } };
+ return { status: 200, body: { access_token: TOKEN, token_type: 'Bearer', expires_in: 300 } };
+ });
+ host.http.set('http://127.0.0.1:4000/internal/operator/alerts', ({ method, headers, body }) => {
+ assert.strictEqual(method, 'POST');
+ assert.strictEqual(headers.Authorization, `Bearer ${TOKEN}`);
+ const doc = JSON.parse(body);
+ host.delivered.push(doc);
+ if (network !== 'ok') return network;
+ return { status: 200, body: { ok: true, firing: doc.alerts.length, opened: doc.alerts.length, reminded: 0, resolved: 0, notified: doc.alerts.length } };
+ });
+ return host;
+}
+
+function noSecrets(host, out) {
+ for (const s of [HOST_SECRET, TOKEN]) {
+ assert.ok(!out.includes(s), 'the output shows no secret');
+ for (const [p, f] of host.files) if (p !== '/etc/openvibe/host.env' && f.type === 'file') assert.ok(!String(f.content).includes(s), `${p} holds no secret`);
+ }
+}
+
+runTests([
+ test('toReport: firing only, the rule severities mapped, annotations clipped, no free-form labels, stable fingerprints', async () => {
+ const r = toReport(FIRING);
+ assert.deepStrictEqual(r.map((a) => [a.name, a.severity, a.service || null]), [['OpenVibeBackupMissed', 'critical', null], ['OpenVibeDeployDrift', 'warning', 'live']]);
+ assert.strictEqual(r[0].started_at, '2026-09-26T05:35:00.123Z');
+ assert.strictEqual(r[1].description.length, 1000);
+ assert.ok(!JSON.stringify(r).includes('127.0.0.1:9100'), 'instance and job labels stay on the host');
+ assert.strictEqual(r[0].fingerprint, fingerprint({ severity: 'page', alertname: 'OpenVibeBackupMissed', job: 'node', instance: '127.0.0.1:9100' }), 'label order does not matter');
+ assert.notStrictEqual(r[0].fingerprint, r[1].fingerprint);
+ const v = contracts.validate('network.operator-alerts-request@1', { source: 'prometheus', alerts: r });
+ assert.ok(v.valid, JSON.stringify(v.errors));
+ assert.deepStrictEqual(toReport([]), []);
+ }),
+ test('alerts relay: a token for network.operator.alert, the complete firing set delivered, metrics written', async () => {
+ const host = withNetwork();
+ const r = await host.cli('alerts', 'relay');
+ assert.strictEqual(r.code, 0, r.out);
+ assert.match(r.out, /ALERTS delivered: 2 firing; opened 2, reminded 0, resolved 0; 2 notification\(s\)/);
+ assert.deepStrictEqual(host.tokenRequests.map((t) => [t.grant_type, t.client_id, t.audience, t.scope]), [['client_credentials', 'host', 'openvibe.network', 'network.operator.alert']]);
+ assert.strictEqual(host.delivered.length, 1);
+ const doc = host.delivered[0];
+ assert.ok(contracts.validate('network.operator-alerts-request@1', doc).valid);
+ assert.deepStrictEqual(doc.alerts.map((a) => a.name), ['OpenVibeBackupMissed', 'OpenVibeDeployDrift']);
+ const prom = host.files.get(PROM_FILE).content;
+ assert.match(prom, /^openvibe_alert_relay_last_run_ok 1$/m);
+ assert.match(prom, /^openvibe_alert_relay_firing 2$/m);
+ assert.match(prom, /^openvibe_alert_relay_last_success_timestamp_seconds \d+$/m);
+ noSecrets(host, r.out);
+ }),
+ test('alerts relay: nothing firing sends an empty set (Network resolves what was open)', async () => {
+ const host = withNetwork({ prom: { status: 200, body: { status: 'success', data: { alerts: [] } } } });
+ const r = await host.cli('alerts', 'relay');
+ assert.strictEqual(r.code, 0, r.out);
+ assert.deepStrictEqual(host.delivered.map((d) => d.alerts), [[]]);
+ }),
+ test('alerts relay: Prometheus down or odd → nothing is sent (an empty set would resolve everything), exit 2', async () => {
+ for (const prom of [null, { status: 503, body: 'no' }, { status: 200, body: { status: 'error' } }]) {
+ const host = withNetwork({ prom });
+ const r = await host.cli('alerts', 'relay');
+ assert.strictEqual(r.code, 2, r.out);
+ assert.match(r.out, /ALERTS not delivered \(prometheus\)/);
+ assert.strictEqual(host.delivered.length, 0);
+ assert.strictEqual(host.tokenRequests.length, 0);
+ assert.match(host.files.get(PROM_FILE).content, /^openvibe_alert_relay_last_run_ok 0$/m);
+ assert.match(host.files.get(PROM_FILE).content, /^openvibe_alert_relay_firing NaN$/m);
+ }
+ }),
+ test('alerts relay: Network refusing → exit 2 with the stage; the last success is kept from the previous file', async () => {
+ const host = withNetwork({ network: { status: 403, body: { error: 'Forbidden' } } });
+ host.put(PROM_FILE, 'openvibe_alert_relay_last_success_timestamp_seconds 1790000000\n');
+ const r = await host.cli('alerts', 'relay');
+ assert.strictEqual(r.code, 2);
+ assert.match(r.out, /ALERTS not delivered \(deliver\): Network answered 403: Forbidden/);
+ assert.match(host.files.get(PROM_FILE).content, /^openvibe_alert_relay_last_success_timestamp_seconds 1790000000$/m);
+ noSecrets(host, r.out);
+ }),
+ test('alerts relay --dry-run: reads and maps, asks for no token, sends nothing; --json', async () => {
+ const host = withNetwork();
+ const r = await host.cli('alerts', 'relay', '--dry-run');
+ assert.strictEqual(r.code, 0);
+ assert.match(r.out, /2 alert\(s\) firing \(dry run, nothing sent\)/);
+ assert.match(r.out, /critical +OpenVibeBackupMissed/);
+ assert.deepStrictEqual([host.tokenRequests.length, host.delivered.length], [0, 0]);
+ const j = await withNetwork().cli('alerts', 'relay', '--json');
+ assert.strictEqual(JSON.parse(j.out).result.opened, 2);
+ }),
+ test('alerts relay: no credentials → exit 2 at the credentials stage', async () => {
+ const host = withNetwork();
+ host.files.delete('/etc/openvibe/host.env');
+ const r = await host.cli('alerts', 'relay');
+ assert.strictEqual(r.code, 2);
+ assert.match(r.out, /ALERTS not delivered \(credentials\)/);
+ }),
+ test('alerts: usage', async () => {
+ const r = await scenario().cli('alerts');
+ assert.notStrictEqual(r.code, 0);
+ assert.match(r.out, /usage: ovhost alerts relay/);
+ }),
+]);