openvibers/openvibe.host Commits
The release UX check (roadmap WS-H task 11): ovhost browser-watch --sites live (openvibe-browsercheck.timer, every 5 minutes) checks each watched site in Chrome after every release (read from /release.json) and once a day. Chrome runs as the unprivileged ovcheck account through runuser, and a failure is confirmed by a second run. The outcome goes to openvibe_browsercheck.prom: OpenVibeBrowserCheckFailed (page) goes through ovhost alerts relay to the operator, and OpenVibeBrowserCheckMissed notices when no check finished for 26 hours. A run that broke records nothing and is retried. The browser-check hook gains asUser/home and returns stdout. docs/operator-alerts.md; test/browser-watch.test.js
Changes
@@@ -79,6 +79,20 @@ groups:labels: { severity: ticket }annotations:summary: "No passing Tools job proof for more than 13 hours (two runs)"++ # The release UX check (ovhost browser-watch, openvibe-browsercheck.timer; WS-H task 11): each watched site++ # in a real browser after every release and daily, as ovcheck; a failure is confirmed by a second run.++ - alert: OpenVibeBrowserCheckFailed++ expr: openvibe_browser_check_ok == 0++ for: 1m++ labels: { severity: page }++ annotations:++ summary: "{{ $labels.service }}: the browser check of release {{ $labels.release }} failed ({{ with printf \"openvibe_browser_check_failing_checks{service='%s'}\" $labels.service | query }}{{ . | first | value }}{{ end }} failing check(s)); run: sudo ovhost browser-watch --sites {{ $labels.service }} --force"++ - alert: OpenVibeBrowserCheckMissed++ expr: time() - openvibe_browser_check_last_run_timestamp_seconds > 26 * 3600++ for: 30m++ labels: { severity: ticket }++ annotations:++ summary: "{{ $labels.service }}: no browser check has run to the end for more than 26 hours"# The relay itself (openvibe-alerts.timer; WS-H task 11). It cannot page while it is broken; it pages# this as soon as a run gets through again, so an outage of the pager is never silent afterwards.- alert: OpenVibeAlertRelayFailingnnew file mode 100644@@@ -0,0 +1,18 @@++[Unit]++Description=OpenVibe release UX check: each watched site in a real browser after every release (WS-H task 11)++Documentation=https://github.com/OpenVibers/OpenVibe.Host/blob/main/docs/operator-alerts.md++After=network-online.target++Wants=network-online.target++++[Service]++Type=oneshot++# Root reads the state and writes the textfile metrics; Chrome runs as the unprivileged ovcheck account++# (runuser), never as root. --sites names the watched services (inventory origin or manifest publicOrigin).++User=root++ExecStart=/usr/local/bin/ovhost browser-watch --sites live++Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin++Environment=OVHOST_INVENTORY=/etc/openvibe/host.json++Nice=10++TimeoutStartSec=30min++PrivateTmp=true++ProtectSystem=fullnnew file mode 100644@@@ -0,0 +1,10 @@++[Unit]++Description=Check each watched OpenVibe site in a browser after a release (every five minutes; WS-H task 11)++++[Timer]++OnBootSec=5min++OnUnitActiveSec=5min++AccuracySec=30s++++[Install]++WantedBy=timers.target@@@ -31,11 +31,32 @@ Prometheus (127.0.0.1:9090) ──GET /api/v1/alerts──▶ ovhost alerts re- **Relay failures**: `openvibe_alert_relay_last_run_ok` and `openvibe_alert_relay_last_success_timestamp_seconds`record them, and `OpenVibeAlertRelayFailing` pages as soon as delivery works again.++## The release UX check++++`ovhost browser-watch --sites live` (openvibe-browsercheck.timer, every 5 minutes) reads each watched site's++`/release.json`. For a release it has not checked yet, or once a day, it runs++[scripts/browser-check.js](../scripts/browser-check.js) in Chrome as the unprivileged `ovcheck` account (never++root): routes × widths, console errors, overflow, duplicate scripts, no-JS content, axe, navigation growth.++++- **Confirmation**: a failure is re-run once before it counts.++- **Outcome**: the result is kept in `/var/lib/openvibe-host/browser-watch/<site>.json` and written to++ `openvibe_browsercheck.prom`.++- **Paging**: `OpenVibeBrowserCheckFailed` (severity page) pages through the relay, naming the release and the++ number of failing checks.++- **Runs that never finish**: a run that broke (exit 2, e.g. Chrome missing) records nothing and is retried++ on the next tick, and `OpenVibeBrowserCheckMissed` notices a site with no finished check for 26 hours.++- **Cost**: a check of openvibe.live takes about a minute on the host.++- **Setup**: Chrome comes from Google's apt repository (`google-chrome-stable`, updated with the system), and++ `ovcheck` is a system account whose home is `/var/lib/ovcheck`.++++To watch more sites, add them to `--sites` in the unit.++## Commands```bashsudo ovhost alerts relay --dry-run # what is firing, nothing sentsudo ovhost alerts relay # deliver now++sudo ovhost browser-watch --sites live --force # check the current release nowsystemctl list-timers openvibe-alerts.timer```@@@ -43,7 +64,13 @@ systemctl list-timers openvibe-alerts.timer```bashsudo cp /usr/local/lib/openvibe-host/deploy/systemd/openvibe-alerts.{service,timer} /etc/systemd/system/--sudo systemctl daemon-reload && sudo systemctl enable --now openvibe-alerts.timer++sudo cp /usr/local/lib/openvibe-host/deploy/systemd/openvibe-browsercheck.{service,timer} /etc/systemd/system/++sudo systemctl daemon-reload && sudo systemctl enable --now openvibe-alerts.timer openvibe-browsercheck.timer++# Chrome and the check account (once):++curl -fsSL https://dl.google.com/linux/linux_signing_key.pub | sudo gpg --dearmor -o /usr/share/keyrings/google-chrome.gpg++echo "deb [arch=amd64 signed-by=/usr/share/keyrings/google-chrome.gpg] https://dl.google.com/linux/chrome/deb/ stable main" | sudo tee /etc/apt/sources.list.d/google-chrome.list++sudo apt-get update && sudo apt-get install -y google-chrome-stable++sudo useradd --system --home-dir /var/lib/ovcheck --create-home --shell /usr/sbin/nologin ovchecksudo cp /usr/local/lib/openvibe-host/deploy/prometheus/openvibe-rules.yml /etc/prometheus/rules/openvibe.ymlsudo promtool check rules /etc/prometheus/rules/openvibe.yml && sudo systemctl reload prometheus```@@@ -10,10 +10,17 @@ const path = require('path');const SCRIPT = path.join(__dirname, '..', 'scripts', 'browser-check.js');--/** Answers { code, lines, error }: code 0 pass, 1 problems found, 2 (or a signal) the check did not run. */--function runBrowserCheck(service, { timeoutMs = 15 * 60 * 1000, script = SCRIPT, args = [] } = {}) {++/**++ * Answers { code, lines, stdout, error }: code 0 pass, 1 problems found, 2 (or a signal) the check did not run.++ * asUser runs it as that account through runuser (Chrome never runs as root; ovhost browser-watch uses ovcheck),++ * with HOME set to home.++ */++function runBrowserCheck(service, { timeoutMs = 15 * 60 * 1000, script = SCRIPT, args = [], asUser = null, home = null, spawnFn = spawn } = {}) {return new Promise((resolve) => {-- const child = spawn(process.execPath, [script, '--sites', service, ...args], { stdio: ['ignore', 'pipe', 'pipe'] });++ const argv = [script, '--sites', service, ...args];++ const child = asUser++ ? spawnFn('runuser', ['-u', asUser, '--', 'env', `HOME=${home || '/tmp'}`, process.execPath, ...argv], { stdio: ['ignore', 'pipe', 'pipe'] })++ : spawnFn(process.execPath, argv, { stdio: ['ignore', 'pipe', 'pipe'] });let out = '', err = '';child.stdout.on('data', (c) => { out += c; });child.stderr.on('data', (c) => { err += c; });@@@ -24,7 +31,7 @@ function runBrowserCheck(service, { timeoutMs = 15 * 60 * 1000, script = SCRIPT,const lines = out.split('\n').map((l) => l.trim()).filter(Boolean);const errLines = err.split('\n').map((l) => l.trim()).filter(Boolean);const c = code == null ? 2 : code;-- resolve({ code: c, lines, error: c >= 2 ? (errLines.find((l) => /Error|not found|timed out/.test(l)) || errLines[errLines.length - 1] || signal || `exit ${c}`) : null });++ resolve({ code: c, lines, stdout: out, error: c >= 2 ? (errLines.find((l) => /Error|not found|timed out/.test(l)) || errLines[errLines.length - 1] || signal || `exit ${c}`) : null });});});}nnew file mode 100644@@@ -0,0 +1,124 @@++'use strict';++/**++ * `ovhost browser-watch` (roadmap WS-H task 11, "release UX check"): each watched site is checked in a real++ * browser (scripts/browser-check.js through lib/browser-check-hook.js) after every release, and once a day++ * without one, and a failure pages the operator (OpenVibeBrowserCheckFailed → ovhost alerts relay → Network).++ *++ * ovhost browser-watch [--sites live,network] [--force] (openvibe-browsercheck.timer: every 5 minutes)++ *++ * Per site: GET <origin>/release.json. A release not checked yet, or a last check older than 24 hours, runs the++ * check as the unprivileged `ovcheck` account (Chrome never runs as root). A failing check runs once more before++ * it counts (a single network blip is not a regression). The outcome is kept in++ * <stateDir>/browser-watch/<site>.json, and every site's latest outcome is written to the textfile collector++ * (openvibe_browsercheck.prom):++ *++ * openvibe_browser_check_ok{service,release} 1 passed, 0 failed++ * openvibe_browser_check_failing_checks{service} checks with failures (status, console, overflow, axe…)++ * openvibe_browser_check_last_run_timestamp_seconds{service} when the last check ran to the end++ *++ * A check that could not run (Chrome missing, the harness broke: exit 2) is not recorded as a pass or a fail;++ * the next timer run tries again, and OpenVibeBrowserCheckMissed notices when none has run for a day.++ * The origin comes from the inventory entry's `origin`, else the service manifest's publicOrigin.++ */++const path = require('path');++const { runBrowserCheck } = require('./browser-check-hook');++++const DAY_MS = 24 * 3600 * 1000;++const TEXTFILE_DIR = '/var/lib/prometheus/node-exporter';++const SITE_RE = /^[a-z][a-z0-9-]{1,39}$/;++const RELEASE_RE = /^[0-9A-Za-z._+-]{1,64}$/;++++function originOf(inv, id, contracts) {++ const entry = inv.services && inv.services[id];++ if (entry && entry.origin) return String(entry.origin).replace(/\/+$/, '');++ let lib = contracts;++ if (!lib) { try { lib = require('openvibe-contracts'); } catch { return null; } }++ const m = lib.services && typeof lib.services.get === 'function' ? lib.services.get(id) : null;++ return m && m.publicOrigin ? String(m.publicOrigin).replace(/\/+$/, '') : null;++}++++async function currentRelease(exec, origin) {++ const r = await exec.request(`${origin}/release.json`, { timeoutMs: 8000 });++ if (r.status !== 200) return null;++ try {++ const m = JSON.parse(r.body);++ const rel = String(m.release || m.commit || '').slice(0, 12);++ return RELEASE_RE.test(rel) ? rel : null;++ } catch { return null; }++}++++/** The failing checks of a --json report list: [{ check, fail }]. */++function failingChecks(reports) {++ const out = [];++ for (const r of reports || []) {++ const checks = (r.summary && r.summary.checks) || {};++ for (const [k, c] of Object.entries(checks)) if (c && c.fail) out.push({ check: k, fail: c.fail });++ if (r.error) out.push({ check: 'run', fail: 1 });++ }++ return out;++}++++const esc = (v) => String(v).replace(/\\/g, '\\\\').replace(/"/g, '\\"');++++async function writeMetrics(exec, inv) {++ const dir = inv.textfileDir || TEXTFILE_DIR;++ if (!(await exec.readdir(dir))) return false;++ const stateDir = path.join(inv.stateDir, 'browser-watch');++ const rows = [];++ for (const e of (await exec.readdir(stateDir)) || []) {++ if (!e.name.endsWith('.json')) continue;++ try { const s = JSON.parse((await exec.readFile(path.join(stateDir, e.name))) || 'null'); if (s && SITE_RE.test(s.site)) rows.push(s); } catch { /* skip */ }++ }++ rows.sort((a, b) => a.site.localeCompare(b.site));++ const lines = [++ '# HELP openvibe_browser_check_ok 1 when the last browser check of the site passed (label: the release it checked).',++ '# TYPE openvibe_browser_check_ok gauge',++ ...rows.map((s) => `openvibe_browser_check_ok{service="${esc(s.site)}",release="${esc(s.release || 'unknown')}"} ${s.ok ? 1 : 0}`),++ '# HELP openvibe_browser_check_failing_checks Checks with failures in the last browser check of the site.',++ '# TYPE openvibe_browser_check_failing_checks gauge',++ ...rows.map((s) => `openvibe_browser_check_failing_checks{service="${esc(s.site)}"} ${(s.failing || []).length}`),++ '# HELP openvibe_browser_check_last_run_timestamp_seconds When the last browser check of the site ran to the end.',++ '# TYPE openvibe_browser_check_last_run_timestamp_seconds gauge',++ ...rows.map((s) => `openvibe_browser_check_last_run_timestamp_seconds{service="${esc(s.site)}"} ${Math.floor(Date.parse(s.checkedAt) / 1000)}`),++ ];++ await exec.writeFile(path.join(dir, 'openvibe_browsercheck.prom'), `${lines.join('\n')}\n`, { mode: 0o644 });++ return true;++}++++/**++ * One pass over the sites. Answers { sites: [{ site, release, action: 'checked'|'skipped'|'error', ok?, failing?, reason? }] }.++ * opts: sites (ids), force, asUser (default ovcheck), home, now, check (the runBrowserCheck to use; tests).++ */++async function watch(ctx, { sites = ['live'], force = false, asUser = 'ovcheck', home = '/var/lib/ovcheck', now = Date.now(), check = runBrowserCheck, contracts = null } = {}) {++ const { exec, inv } = ctx;++ const stateDir = path.join(inv.stateDir, 'browser-watch');++ await exec.mkdir(stateDir, { mode: 0o750 });++ const out = [];++ for (const site of sites) {++ if (!SITE_RE.test(site)) { out.push({ site, action: 'error', reason: 'not a service id' }); continue; }++ const origin = originOf(inv, site, contracts);++ if (!origin) { out.push({ site, action: 'error', reason: 'no origin (inventory origin or manifest publicOrigin)' }); continue; }++ const release = await currentRelease(exec, origin);++ const file = path.join(stateDir, `${site}.json`);++ let prev = null;++ try { prev = JSON.parse((await exec.readFile(file)) || 'null'); } catch { prev = null; }++ const fresh = prev && prev.release === release && now - Date.parse(prev.checkedAt) < DAY_MS;++ if (fresh && !force) { out.push({ site, release, action: 'skipped', ok: prev.ok, reason: 'this release was checked' }); continue; }++ let r = null;++ let reports = null;++ for (let attempt = 1; attempt <= 2; attempt++) {++ r = await check(site, { args: ['--json'], asUser, home });++ try { reports = JSON.parse(r.stdout || 'null'); } catch { reports = null; }++ if (r.code === 0 || r.code >= 2 || !reports) break; // a pass, or a run that broke: no second attempt++ }++ if (r.code >= 2 || !Array.isArray(reports)) { out.push({ site, release, action: 'error', reason: r.error || `the check did not run (exit ${r.code})` }); continue; }++ const failing = failingChecks(reports);++ const state = { site, origin, release, ok: r.code === 0, failing, checkedAt: new Date(now).toISOString() };++ await exec.writeFile(file, `${JSON.stringify(state, null, 2)}\n`, { mode: 0o640 });++ out.push({ site, release, action: 'checked', ok: state.ok, failing });++ }++ try { await writeMetrics(exec, inv); } catch { /* metrics never fail a run */ }++ return { sites: out };++}++++module.exports = { watch, failingChecks, writeMetrics, originOf, currentRelease, DAY_MS };@@@ -62,6 +62,10 @@ Usage: ovhost <command> [args] [--inventory <file>] [--json]declared by the inventory) to <prefix>-archive/; never deletes themarchive list the archives off-hostarchive restore <stamp> <path> --out <dir> one archived file back, checked against the manifest++ browser-watch [--sites live,...] [--force]++ check each site in a real browser after every release (and++ daily) as ovcheck; metrics for OpenVibeBrowserCheckFailed++ (openvibe-browsercheck.timer, every 5 minutes)alerts relay [--prometheus <url>] [--dry-run]the alerts firing in Prometheus, sent to Network, which pagesthe operator (openvibe-alerts.timer, every 2 minutes)@@@ -83,7 +87,7 @@ drill: 0 passed · 1 precondition (not root, port in use, no backup, unsupportedbackup --all / offsite: 0 everything succeeded · 1 precondition (config, key, lock) · 2 something failed`;--const FLAGS_WITH_VALUE = new Set(['prometheus', 'inventory', 'to', 'limit', 'warn-days', 'variant', 'manifest', 'out', 'ready-timeout', 'wildcard-cert', 'backup', 'run', 'keep-daily', 'keep-weekly', 'from-host', 'backup-env', 'release', 'commit', 'origin', 'events-env', 'note']);++const FLAGS_WITH_VALUE = new Set(['prometheus', 'sites', 'inventory', 'to', 'limit', 'warn-days', 'variant', 'manifest', 'out', 'ready-timeout', 'wildcard-cert', 'backup', 'run', 'keep-daily', 'keep-weekly', 'from-host', 'backup-env', 'release', 'commit', 'origin', 'events-env', 'note']);function parseArgs(argv) {const pos = [];@@@ -370,6 +374,15 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => procelse out(`OFFSITE ${o.ok ? 'ok' : 'FAILED'}: ${o.uploaded.length} file(s) to s3://${o.bucket}/${o.location}${o.failures.length ? `; failures: ${o.failures.map((f) => `${f.service ? `${f.service}/` : ''}${f.file}: ${f.error}`).join('; ')}` : ''}`);return o.ok ? 0 : 2;}++ case 'browser-watch': {++ // WS-H task 11: the release UX check. A failure pages through the alerts relay.++ const sites = String(opts.sites || 'live').split(',').map((x) => x.trim()).filter(Boolean);++ const r = await lib.browserWatch.watch(ctx, { sites, force: !!opts.force });++ if (json) emit(r);++ else for (const x of r.sites) out(x.action === 'checked' ? `${x.ok ? 'ok ' : 'FAIL'} ${x.site} ${x.release || '?'}${x.failing && x.failing.length ? `; failing: ${x.failing.map((f) => `${f.check} ${f.fail}`).join(', ')}` : ''}`++ : x.action === 'skipped' ? `skip ${x.site} ${x.release || '?'} (${x.reason}; last ${x.ok ? 'passed' : 'failed'})` : `ERR ${x.site}: ${x.reason}`);++ return r.sites.some((x) => x.action === 'error') ? 2 : r.sites.some((x) => x.action === 'checked' && !x.ok) ? 1 : 0;++ }case 'alerts': {// WS-H task 11: deliver the firing Prometheus alerts to the operator through Network.if (pos[1] !== 'relay') throw new UsageError('usage: ovhost alerts relay [--prometheus <url>] [--dry-run] [--events-env <file>]');@@@ -35,6 +35,7 @@ module.exports = {offsite,archive: require('./archive'),alerts: require('./alerts'),++ browserWatch: require('./browser-watch'),snapshot,drill,DrillError,@@@ -173,9 +173,9 @@ runTests([try {const script = path.join(dir, 'check.js');fs.writeFileSync(script, "const a = process.argv.slice(2); if (a[1] === 'down') { console.error('Error: Chrome not found (set CHROME_BIN)'); process.exit(2); } console.log(`ok site ${a.join(' ')}`); process.exit(a[1] === 'bad' ? 1 : 0);");-- assert.deepStrictEqual(await runBrowserCheck('wiki', { script, args: ['--no-nav'] }), { code: 0, lines: ['ok site --sites wiki --no-nav'], error: null });++ assert.deepStrictEqual(await runBrowserCheck('wiki', { script, args: ['--no-nav'] }), { code: 0, lines: ['ok site --sites wiki --no-nav'], stdout: 'ok site --sites wiki --no-nav\n', error: null });assert.strictEqual((await runBrowserCheck('bad', { script })).code, 1);-- assert.deepStrictEqual(await runBrowserCheck('down', { script }), { code: 2, lines: [], error: 'Error: Chrome not found (set CHROME_BIN)' });++ assert.deepStrictEqual(await runBrowserCheck('down', { script }), { code: 2, lines: [], stdout: '', error: 'Error: Chrome not found (set CHROME_BIN)' });fs.writeFileSync(script, 'setTimeout(() => {}, 60000);');const slow = await runBrowserCheck('x', { script, timeoutMs: 300 });assert.strictEqual(slow.code, 2);nnew file mode 100644@@@ -0,0 +1,89 @@++'use strict';++/**++ * The release UX check (lib/browser-watch.js, WS-H task 11): a site is checked in a browser when its release++ * changes and once a day, as ovcheck; a failure is confirmed by a second run; a run that broke records++ * nothing; the outcome reaches the textfile collector for OpenVibeBrowserCheckFailed.++ */++const assert = require('assert');++const { spawn } = require('child_process');++const { scenario, test, runTests } = require('./helpers');++const { watch, failingChecks, DAY_MS } = require('../lib/browser-watch');++const { runBrowserCheck } = require('../lib/browser-check-hook');++++const PROM = '/var/lib/prometheus/node-exporter/openvibe_browsercheck.prom';++const pass = [{ base: 'https://openvibe.live', ok: true, summary: { checks: { status: { fail: 0 }, console: { fail: 0 } } } }];++const fail = [{ base: 'https://openvibe.live', ok: false, summary: { checks: { status: { fail: 0 }, console: { fail: 2 }, overflow: { fail: 1 } } } }];++++function site({ release = 'abc1234def56' } = {}) {++ const host = scenario();++ host.inv.services.live.origin = 'https://openvibe.live';++ host.put('/var/lib/prometheus/node-exporter/.keep', '');++ host.release = release;++ host.http.set('https://openvibe.live/release.json', () => ({ status: 200, body: { service: 'live', release: host.release } }));++ host.runs = [];++ host.answers = [];++ host.check = async (s, opts) => { host.runs.push({ s, opts }); const a = host.answers.shift() || { code: 0, reports: pass }; return { code: a.code, stdout: a.reports ? JSON.stringify(a.reports) : '', error: a.error || null }; };++ return host;++}++++runTests([++ test('a new release is checked as ovcheck with --json; the same release is skipped; a day later it is checked again', async () => {++ const host = site();++ const now = Date.parse('2026-09-26T14:00:00Z');++ let r = await watch(host.ctx(), { check: host.check, now });++ assert.deepStrictEqual(r.sites.map((x) => [x.site, x.release, x.action, x.ok]), [['live', 'abc1234def56', 'checked', true]]);++ assert.deepStrictEqual(host.runs.map((x) => [x.s, x.opts.asUser, x.opts.home, x.opts.args]), [['live', 'ovcheck', '/var/lib/ovcheck', ['--json']]]);++ assert.match(host.files.get(PROM).content, /^openvibe_browser_check_ok\{service="live",release="abc1234def56"\} 1$/m);++ assert.match(host.files.get(PROM).content, /^openvibe_browser_check_failing_checks\{service="live"\} 0$/m);++ r = await watch(host.ctx(), { check: host.check, now: now + 60000 });++ assert.strictEqual(r.sites[0].action, 'skipped');++ assert.strictEqual(host.runs.length, 1);++ r = await watch(host.ctx(), { check: host.check, now: now + DAY_MS + 1 });++ assert.strictEqual(r.sites[0].action, 'checked', 'daily even without a release');++ host.release = 'fff0000aaa11';++ r = await watch(host.ctx(), { check: host.check, now: now + DAY_MS + 60000 });++ assert.deepStrictEqual([r.sites[0].action, r.sites[0].release], ['checked', 'fff0000aaa11']);++ r = await watch(host.ctx(), { check: host.check, now: now + DAY_MS + 120000, force: true });++ assert.strictEqual(r.sites[0].action, 'checked', '--force');++ }),++ test('a failure is confirmed by a second run before it counts; one flake then a pass is a pass', async () => {++ let host = site();++ host.answers.push({ code: 1, reports: fail }, { code: 1, reports: fail });++ let r = await watch(host.ctx(), { check: host.check });++ assert.strictEqual(host.runs.length, 2);++ assert.deepStrictEqual([r.sites[0].ok, r.sites[0].failing], [false, [{ check: 'console', fail: 2 }, { check: 'overflow', fail: 1 }]]);++ assert.match(host.files.get(PROM).content, /^openvibe_browser_check_ok\{service="live",release="abc1234def56"\} 0$/m);++ assert.match(host.files.get(PROM).content, /^openvibe_browser_check_failing_checks\{service="live"\} 2$/m);++ host = site();++ host.answers.push({ code: 1, reports: fail }, { code: 0, reports: pass });++ r = await watch(host.ctx(), { check: host.check });++ assert.deepStrictEqual([host.runs.length, r.sites[0].ok], [2, true]);++ }),++ test('a run that broke (exit 2) records nothing and is retried by the next timer run', async () => {++ const host = site();++ host.answers.push({ code: 2, error: 'Chrome not found' });++ let r = await watch(host.ctx(), { check: host.check });++ assert.deepStrictEqual([r.sites[0].action, r.sites[0].reason, host.runs.length], ['error', 'Chrome not found', 1]);++ assert.ok(!host.files.has('/var/lib/openvibe-host/browser-watch/live.json'));++ r = await watch(host.ctx(), { check: host.check });++ assert.strictEqual(r.sites[0].action, 'checked');++ }),++ test('bad ids and sites without an origin are errors, not checks; the CLI exits 1 on a failed check and 2 on an error', async () => {++ const host = site();++ const r = await watch(host.ctx(), { check: host.check, sites: ['Bad!', 'nosuchservice'], contracts: { services: { get: () => null } } });++ assert.deepStrictEqual(r.sites.map((x) => x.action), ['error', 'error']);++ assert.strictEqual(host.runs.length, 0);++ const c = await scenario().cli('browser-watch', '--sites', 'Bad!');++ assert.strictEqual(c.code, 2);++ assert.match(c.out, /ERR {2}Bad!: not a service id/);++ }),++ test('failingChecks reads the reports; the hook runs as another user through runuser with HOME', async () => {++ assert.deepStrictEqual(failingChecks([{ summary: { checks: { a: { fail: 0 }, b: { fail: 3 } } } }, { error: 'boom', summary: { checks: {} } }]), [{ check: 'b', fail: 3 }, { check: 'run', fail: 1 }]);++ const seen = [];++ const fake = (cmd, args) => { seen.push([cmd, ...args]); return spawn(process.execPath, ['-e', 'process.stdout.write("[]")']); };++ const r = await runBrowserCheck('live', { args: ['--json'], asUser: 'ovcheck', home: '/var/lib/ovcheck', spawnFn: fake });++ assert.deepStrictEqual([r.code, r.stdout], [0, '[]']);++ assert.deepStrictEqual(seen[0].slice(0, 6), ['runuser', '-u', 'ovcheck', '--', 'env', 'HOME=/var/lib/ovcheck']);++ assert.ok(seen[0].includes('--sites') && seen[0].includes('live') && seen[0].includes('--json'));++ }),++]);
@@ -79,6 +79,20 @@ groups:
@@ -79,6 +79,20 @@ groups:
labels: { severity: ticket }
labels: { severity: ticket }
annotations:
annotations:
summary: "No passing Tools job proof for more than 13 hours (two runs)"
summary: "No passing Tools job proof for more than 13 hours (two runs)"
+ # The release UX check (ovhost browser-watch, openvibe-browsercheck.timer; WS-H task 11): each watched site
+ # in a real browser after every release and daily, as ovcheck; a failure is confirmed by a second run.
+ - alert: OpenVibeBrowserCheckFailed
+ expr: openvibe_browser_check_ok == 0
+ for: 1m
+ labels: { severity: page }
+ annotations:
+ summary: "{{ $labels.service }}: the browser check of release {{ $labels.release }} failed ({{ with printf \"openvibe_browser_check_failing_checks{service='%s'}\" $labels.service | query }}{{ . | first | value }}{{ end }} failing check(s)); run: sudo ovhost browser-watch --sites {{ $labels.service }} --force"
+ - alert: OpenVibeBrowserCheckMissed
+ expr: time() - openvibe_browser_check_last_run_timestamp_seconds > 26 * 3600
+ for: 30m
+ labels: { severity: ticket }
+ annotations:
+ summary: "{{ $labels.service }}: no browser check has run to the end for more than 26 hours"
# The relay itself (openvibe-alerts.timer; WS-H task 11). It cannot page while it is broken; it pages
# The relay itself (openvibe-alerts.timer; WS-H task 11). It cannot page while it is broken; it pages
# this as soon as a run gets through again, so an outage of the pager is never silent afterwards.
# this as soon as a run gets through again, so an outage of the pager is never silent afterwards.
- alert: OpenVibeAlertRelayFailing
- alert: OpenVibeAlertRelayFailing
new file mode 100644
new file mode 100644
@@ -0,0 +1,18 @@
@@ -0,0 +1,18 @@
+[Unit]
+Description=OpenVibe release UX check: each watched site in a real browser after every release (WS-H task 11)
+Documentation=https://github.com/OpenVibers/OpenVibe.Host/blob/main/docs/operator-alerts.md
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+# Root reads the state and writes the textfile metrics; Chrome runs as the unprivileged ovcheck account
+# (runuser), never as root. --sites names the watched services (inventory origin or manifest publicOrigin).
+User=root
+ExecStart=/usr/local/bin/ovhost browser-watch --sites live
+Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+Environment=OVHOST_INVENTORY=/etc/openvibe/host.json
+Nice=10
+TimeoutStartSec=30min
+PrivateTmp=true
+ProtectSystem=full
new file mode 100644
new file mode 100644
@@ -0,0 +1,10 @@
@@ -0,0 +1,10 @@
+[Unit]
+Description=Check each watched OpenVibe site in a browser after a release (every five minutes; WS-H task 11)
+
+[Timer]
+OnBootSec=5min
+OnUnitActiveSec=5min
+AccuracySec=30s
+
+[Install]
+WantedBy=timers.target
@@ -31,11 +31,32 @@ Prometheus (127.0.0.1:9090) ──GET /api/v1/alerts──▶ ovhost alerts re
@@ -31,11 +31,32 @@ Prometheus (127.0.0.1:9090) ──GET /api/v1/alerts──▶ ovhost alerts re
- **Relay failures**: `openvibe_alert_relay_last_run_ok` and `openvibe_alert_relay_last_success_timestamp_seconds`
- **Relay failures**: `openvibe_alert_relay_last_run_ok` and `openvibe_alert_relay_last_success_timestamp_seconds`
record them, and `OpenVibeAlertRelayFailing` pages as soon as delivery works again.
record them, and `OpenVibeAlertRelayFailing` pages as soon as delivery works again.
+## The release UX check
+
+`ovhost browser-watch --sites live` (openvibe-browsercheck.timer, every 5 minutes) reads each watched site's
+`/release.json`. For a release it has not checked yet, or once a day, it runs
+[scripts/browser-check.js](../scripts/browser-check.js) in Chrome as the unprivileged `ovcheck` account (never
+root): routes × widths, console errors, overflow, duplicate scripts, no-JS content, axe, navigation growth.
+
+- **Confirmation**: a failure is re-run once before it counts.
+- **Outcome**: the result is kept in `/var/lib/openvibe-host/browser-watch/<site>.json` and written to
+ `openvibe_browsercheck.prom`.
+- **Paging**: `OpenVibeBrowserCheckFailed` (severity page) pages through the relay, naming the release and the
+ number of failing checks.
+- **Runs that never finish**: a run that broke (exit 2, e.g. Chrome missing) records nothing and is retried
+ on the next tick, and `OpenVibeBrowserCheckMissed` notices a site with no finished check for 26 hours.
+- **Cost**: a check of openvibe.live takes about a minute on the host.
+- **Setup**: Chrome comes from Google's apt repository (`google-chrome-stable`, updated with the system), and
+ `ovcheck` is a system account whose home is `/var/lib/ovcheck`.
+
+To watch more sites, add them to `--sites` in the unit.
+
## Commands
## Commands
```bash
```bash
sudo ovhost alerts relay --dry-run # what is firing, nothing sent
sudo ovhost alerts relay --dry-run # what is firing, nothing sent
sudo ovhost alerts relay # deliver now
sudo ovhost alerts relay # deliver now
+sudo ovhost browser-watch --sites live --force # check the current release now
systemctl list-timers openvibe-alerts.timer
systemctl list-timers openvibe-alerts.timer
```
```
@@ -43,7 +64,13 @@ systemctl list-timers openvibe-alerts.timer
@@ -43,7 +64,13 @@ systemctl list-timers openvibe-alerts.timer
```bash
```bash
sudo cp /usr/local/lib/openvibe-host/deploy/systemd/openvibe-alerts.{service,timer} /etc/systemd/system/
sudo cp /usr/local/lib/openvibe-host/deploy/systemd/openvibe-alerts.{service,timer} /etc/systemd/system/
-sudo systemctl daemon-reload && sudo systemctl enable --now openvibe-alerts.timer
+sudo cp /usr/local/lib/openvibe-host/deploy/systemd/openvibe-browsercheck.{service,timer} /etc/systemd/system/
+sudo systemctl daemon-reload && sudo systemctl enable --now openvibe-alerts.timer openvibe-browsercheck.timer
+# Chrome and the check account (once):
+curl -fsSL https://dl.google.com/linux/linux_signing_key.pub | sudo gpg --dearmor -o /usr/share/keyrings/google-chrome.gpg
+echo "deb [arch=amd64 signed-by=/usr/share/keyrings/google-chrome.gpg] https://dl.google.com/linux/chrome/deb/ stable main" | sudo tee /etc/apt/sources.list.d/google-chrome.list
+sudo apt-get update && sudo apt-get install -y google-chrome-stable
+sudo useradd --system --home-dir /var/lib/ovcheck --create-home --shell /usr/sbin/nologin ovcheck
sudo cp /usr/local/lib/openvibe-host/deploy/prometheus/openvibe-rules.yml /etc/prometheus/rules/openvibe.yml
sudo cp /usr/local/lib/openvibe-host/deploy/prometheus/openvibe-rules.yml /etc/prometheus/rules/openvibe.yml
sudo promtool check rules /etc/prometheus/rules/openvibe.yml && sudo systemctl reload prometheus
sudo promtool check rules /etc/prometheus/rules/openvibe.yml && sudo systemctl reload prometheus
```
```
@@ -10,10 +10,17 @@ const path = require('path');
@@ -10,10 +10,17 @@ const path = require('path');
const SCRIPT = path.join(__dirname, '..', 'scripts', 'browser-check.js');
const SCRIPT = path.join(__dirname, '..', 'scripts', 'browser-check.js');
-/** Answers { code, lines, error }: code 0 pass, 1 problems found, 2 (or a signal) the check did not run. */
-function runBrowserCheck(service, { timeoutMs = 15 * 60 * 1000, script = SCRIPT, args = [] } = {}) {
+/**
+ * Answers { code, lines, stdout, error }: code 0 pass, 1 problems found, 2 (or a signal) the check did not run.
+ * asUser runs it as that account through runuser (Chrome never runs as root; ovhost browser-watch uses ovcheck),
+ * with HOME set to home.
+ */
+function runBrowserCheck(service, { timeoutMs = 15 * 60 * 1000, script = SCRIPT, args = [], asUser = null, home = null, spawnFn = spawn } = {}) {
return new Promise((resolve) => {
return new Promise((resolve) => {
- const child = spawn(process.execPath, [script, '--sites', service, ...args], { stdio: ['ignore', 'pipe', 'pipe'] });
+ const argv = [script, '--sites', service, ...args];
+ const child = asUser
+ ? spawnFn('runuser', ['-u', asUser, '--', 'env', `HOME=${home || '/tmp'}`, process.execPath, ...argv], { stdio: ['ignore', 'pipe', 'pipe'] })
+ : spawnFn(process.execPath, argv, { stdio: ['ignore', 'pipe', 'pipe'] });
let out = '', err = '';
let out = '', err = '';
child.stdout.on('data', (c) => { out += c; });
child.stdout.on('data', (c) => { out += c; });
child.stderr.on('data', (c) => { err += c; });
child.stderr.on('data', (c) => { err += c; });
@@ -24,7 +31,7 @@ function runBrowserCheck(service, { timeoutMs = 15 * 60 * 1000, script = SCRIPT,
@@ -24,7 +31,7 @@ function runBrowserCheck(service, { timeoutMs = 15 * 60 * 1000, script = SCRIPT,
const lines = out.split('\n').map((l) => l.trim()).filter(Boolean);
const lines = out.split('\n').map((l) => l.trim()).filter(Boolean);
const errLines = err.split('\n').map((l) => l.trim()).filter(Boolean);
const errLines = err.split('\n').map((l) => l.trim()).filter(Boolean);
const c = code == null ? 2 : code;
const c = code == null ? 2 : code;
- resolve({ code: c, lines, error: c >= 2 ? (errLines.find((l) => /Error|not found|timed out/.test(l)) || errLines[errLines.length - 1] || signal || `exit ${c}`) : null });
+ resolve({ code: c, lines, stdout: out, error: c >= 2 ? (errLines.find((l) => /Error|not found|timed out/.test(l)) || errLines[errLines.length - 1] || signal || `exit ${c}`) : null });
});
});
});
});
}
}
new file mode 100644
new file mode 100644
@@ -0,0 +1,124 @@
@@ -0,0 +1,124 @@
+'use strict';
+/**
+ * `ovhost browser-watch` (roadmap WS-H task 11, "release UX check"): each watched site is checked in a real
+ * browser (scripts/browser-check.js through lib/browser-check-hook.js) after every release, and once a day
+ * without one, and a failure pages the operator (OpenVibeBrowserCheckFailed → ovhost alerts relay → Network).
+ *
+ * ovhost browser-watch [--sites live,network] [--force] (openvibe-browsercheck.timer: every 5 minutes)
+ *
+ * Per site: GET <origin>/release.json. A release not checked yet, or a last check older than 24 hours, runs the
+ * check as the unprivileged `ovcheck` account (Chrome never runs as root). A failing check runs once more before
+ * it counts (a single network blip is not a regression). The outcome is kept in
+ * <stateDir>/browser-watch/<site>.json, and every site's latest outcome is written to the textfile collector
+ * (openvibe_browsercheck.prom):
+ *
+ * openvibe_browser_check_ok{service,release} 1 passed, 0 failed
+ * openvibe_browser_check_failing_checks{service} checks with failures (status, console, overflow, axe…)
+ * openvibe_browser_check_last_run_timestamp_seconds{service} when the last check ran to the end
+ *
+ * A check that could not run (Chrome missing, the harness broke: exit 2) is not recorded as a pass or a fail;
+ * the next timer run tries again, and OpenVibeBrowserCheckMissed notices when none has run for a day.
+ * The origin comes from the inventory entry's `origin`, else the service manifest's publicOrigin.
+ */
+const path = require('path');
+const { runBrowserCheck } = require('./browser-check-hook');
+
+const DAY_MS = 24 * 3600 * 1000;
+const TEXTFILE_DIR = '/var/lib/prometheus/node-exporter';
+const SITE_RE = /^[a-z][a-z0-9-]{1,39}$/;
+const RELEASE_RE = /^[0-9A-Za-z._+-]{1,64}$/;
+
+function originOf(inv, id, contracts) {
+ const entry = inv.services && inv.services[id];
+ if (entry && entry.origin) return String(entry.origin).replace(/\/+$/, '');
+ let lib = contracts;
+ if (!lib) { try { lib = require('openvibe-contracts'); } catch { return null; } }
+ const m = lib.services && typeof lib.services.get === 'function' ? lib.services.get(id) : null;
+ return m && m.publicOrigin ? String(m.publicOrigin).replace(/\/+$/, '') : null;
+}
+
+async function currentRelease(exec, origin) {
+ const r = await exec.request(`${origin}/release.json`, { timeoutMs: 8000 });
+ if (r.status !== 200) return null;
+ try {
+ const m = JSON.parse(r.body);
+ const rel = String(m.release || m.commit || '').slice(0, 12);
+ return RELEASE_RE.test(rel) ? rel : null;
+ } catch { return null; }
+}
+
+/** The failing checks of a --json report list: [{ check, fail }]. */
+function failingChecks(reports) {
+ const out = [];
+ for (const r of reports || []) {
+ const checks = (r.summary && r.summary.checks) || {};
+ for (const [k, c] of Object.entries(checks)) if (c && c.fail) out.push({ check: k, fail: c.fail });
+ if (r.error) out.push({ check: 'run', fail: 1 });
+ }
+ return out;
+}
+
+const esc = (v) => String(v).replace(/\\/g, '\\\\').replace(/"/g, '\\"');
+
+async function writeMetrics(exec, inv) {
+ const dir = inv.textfileDir || TEXTFILE_DIR;
+ if (!(await exec.readdir(dir))) return false;
+ const stateDir = path.join(inv.stateDir, 'browser-watch');
+ const rows = [];
+ for (const e of (await exec.readdir(stateDir)) || []) {
+ if (!e.name.endsWith('.json')) continue;
+ try { const s = JSON.parse((await exec.readFile(path.join(stateDir, e.name))) || 'null'); if (s && SITE_RE.test(s.site)) rows.push(s); } catch { /* skip */ }
+ }
+ rows.sort((a, b) => a.site.localeCompare(b.site));
+ const lines = [
+ '# HELP openvibe_browser_check_ok 1 when the last browser check of the site passed (label: the release it checked).',
+ '# TYPE openvibe_browser_check_ok gauge',
+ ...rows.map((s) => `openvibe_browser_check_ok{service="${esc(s.site)}",release="${esc(s.release || 'unknown')}"} ${s.ok ? 1 : 0}`),
+ '# HELP openvibe_browser_check_failing_checks Checks with failures in the last browser check of the site.',
+ '# TYPE openvibe_browser_check_failing_checks gauge',
+ ...rows.map((s) => `openvibe_browser_check_failing_checks{service="${esc(s.site)}"} ${(s.failing || []).length}`),
+ '# HELP openvibe_browser_check_last_run_timestamp_seconds When the last browser check of the site ran to the end.',
+ '# TYPE openvibe_browser_check_last_run_timestamp_seconds gauge',
+ ...rows.map((s) => `openvibe_browser_check_last_run_timestamp_seconds{service="${esc(s.site)}"} ${Math.floor(Date.parse(s.checkedAt) / 1000)}`),
+ ];
+ await exec.writeFile(path.join(dir, 'openvibe_browsercheck.prom'), `${lines.join('\n')}\n`, { mode: 0o644 });
+ return true;
+}
+
+/**
+ * One pass over the sites. Answers { sites: [{ site, release, action: 'checked'|'skipped'|'error', ok?, failing?, reason? }] }.
+ * opts: sites (ids), force, asUser (default ovcheck), home, now, check (the runBrowserCheck to use; tests).
+ */
+async function watch(ctx, { sites = ['live'], force = false, asUser = 'ovcheck', home = '/var/lib/ovcheck', now = Date.now(), check = runBrowserCheck, contracts = null } = {}) {
+ const { exec, inv } = ctx;
+ const stateDir = path.join(inv.stateDir, 'browser-watch');
+ await exec.mkdir(stateDir, { mode: 0o750 });
+ const out = [];
+ for (const site of sites) {
+ if (!SITE_RE.test(site)) { out.push({ site, action: 'error', reason: 'not a service id' }); continue; }
+ const origin = originOf(inv, site, contracts);
+ if (!origin) { out.push({ site, action: 'error', reason: 'no origin (inventory origin or manifest publicOrigin)' }); continue; }
+ const release = await currentRelease(exec, origin);
+ const file = path.join(stateDir, `${site}.json`);
+ let prev = null;
+ try { prev = JSON.parse((await exec.readFile(file)) || 'null'); } catch { prev = null; }
+ const fresh = prev && prev.release === release && now - Date.parse(prev.checkedAt) < DAY_MS;
+ if (fresh && !force) { out.push({ site, release, action: 'skipped', ok: prev.ok, reason: 'this release was checked' }); continue; }
+ let r = null;
+ let reports = null;
+ for (let attempt = 1; attempt <= 2; attempt++) {
+ r = await check(site, { args: ['--json'], asUser, home });
+ try { reports = JSON.parse(r.stdout || 'null'); } catch { reports = null; }
+ if (r.code === 0 || r.code >= 2 || !reports) break; // a pass, or a run that broke: no second attempt
+ }
+ if (r.code >= 2 || !Array.isArray(reports)) { out.push({ site, release, action: 'error', reason: r.error || `the check did not run (exit ${r.code})` }); continue; }
+ const failing = failingChecks(reports);
+ const state = { site, origin, release, ok: r.code === 0, failing, checkedAt: new Date(now).toISOString() };
+ await exec.writeFile(file, `${JSON.stringify(state, null, 2)}\n`, { mode: 0o640 });
+ out.push({ site, release, action: 'checked', ok: state.ok, failing });
+ }
+ try { await writeMetrics(exec, inv); } catch { /* metrics never fail a run */ }
+ return { sites: out };
+}
+
+module.exports = { watch, failingChecks, writeMetrics, originOf, currentRelease, DAY_MS };
@@ -62,6 +62,10 @@ Usage: ovhost <command> [args] [--inventory <file>] [--json]
@@ -62,6 +62,10 @@ Usage: ovhost <command> [args] [--inventory <file>] [--json]
declared by the inventory) to <prefix>-archive/; never deletes them
declared by the inventory) to <prefix>-archive/; never deletes them
archive list the archives off-host
archive list the archives off-host
archive restore <stamp> <path> --out <dir> one archived file back, checked against the manifest
archive restore <stamp> <path> --out <dir> one archived file back, checked against the manifest
+ browser-watch [--sites live,...] [--force]
+ check each site in a real browser after every release (and
+ daily) as ovcheck; metrics for OpenVibeBrowserCheckFailed
+ (openvibe-browsercheck.timer, every 5 minutes)
alerts relay [--prometheus <url>] [--dry-run]
alerts relay [--prometheus <url>] [--dry-run]
the alerts firing in Prometheus, sent to Network, which pages
the alerts firing in Prometheus, sent to Network, which pages
the operator (openvibe-alerts.timer, every 2 minutes)
the operator (openvibe-alerts.timer, every 2 minutes)
@@ -83,7 +87,7 @@ drill: 0 passed · 1 precondition (not root, port in use, no backup, unsupported
@@ -83,7 +87,7 @@ drill: 0 passed · 1 precondition (not root, port in use, no backup, unsupported
backup --all / offsite: 0 everything succeeded · 1 precondition (config, key, lock) · 2 something failed
backup --all / offsite: 0 everything succeeded · 1 precondition (config, key, lock) · 2 something failed
`;
`;
-const FLAGS_WITH_VALUE = new Set(['prometheus', 'inventory', 'to', 'limit', 'warn-days', 'variant', 'manifest', 'out', 'ready-timeout', 'wildcard-cert', 'backup', 'run', 'keep-daily', 'keep-weekly', 'from-host', 'backup-env', 'release', 'commit', 'origin', 'events-env', 'note']);
+const FLAGS_WITH_VALUE = new Set(['prometheus', 'sites', 'inventory', 'to', 'limit', 'warn-days', 'variant', 'manifest', 'out', 'ready-timeout', 'wildcard-cert', 'backup', 'run', 'keep-daily', 'keep-weekly', 'from-host', 'backup-env', 'release', 'commit', 'origin', 'events-env', 'note']);
function parseArgs(argv) {
function parseArgs(argv) {
const pos = [];
const pos = [];
@@ -370,6 +374,15 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc
@@ -370,6 +374,15 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc
else out(`OFFSITE ${o.ok ? 'ok' : 'FAILED'}: ${o.uploaded.length} file(s) to s3://${o.bucket}/${o.location}${o.failures.length ? `; failures: ${o.failures.map((f) => `${f.service ? `${f.service}/` : ''}${f.file}: ${f.error}`).join('; ')}` : ''}`);
else out(`OFFSITE ${o.ok ? 'ok' : 'FAILED'}: ${o.uploaded.length} file(s) to s3://${o.bucket}/${o.location}${o.failures.length ? `; failures: ${o.failures.map((f) => `${f.service ? `${f.service}/` : ''}${f.file}: ${f.error}`).join('; ')}` : ''}`);
return o.ok ? 0 : 2;
return o.ok ? 0 : 2;
}
}
+ case 'browser-watch': {
+ // WS-H task 11: the release UX check. A failure pages through the alerts relay.
+ const sites = String(opts.sites || 'live').split(',').map((x) => x.trim()).filter(Boolean);
+ const r = await lib.browserWatch.watch(ctx, { sites, force: !!opts.force });
+ if (json) emit(r);
+ else for (const x of r.sites) out(x.action === 'checked' ? `${x.ok ? 'ok ' : 'FAIL'} ${x.site} ${x.release || '?'}${x.failing && x.failing.length ? `; failing: ${x.failing.map((f) => `${f.check} ${f.fail}`).join(', ')}` : ''}`
+ : x.action === 'skipped' ? `skip ${x.site} ${x.release || '?'} (${x.reason}; last ${x.ok ? 'passed' : 'failed'})` : `ERR ${x.site}: ${x.reason}`);
+ return r.sites.some((x) => x.action === 'error') ? 2 : r.sites.some((x) => x.action === 'checked' && !x.ok) ? 1 : 0;
+ }
case 'alerts': {
case 'alerts': {
// WS-H task 11: deliver the firing Prometheus alerts to the operator through Network.
// WS-H task 11: deliver the firing Prometheus alerts to the operator through Network.
if (pos[1] !== 'relay') throw new UsageError('usage: ovhost alerts relay [--prometheus <url>] [--dry-run] [--events-env <file>]');
if (pos[1] !== 'relay') throw new UsageError('usage: ovhost alerts relay [--prometheus <url>] [--dry-run] [--events-env <file>]');
@@ -35,6 +35,7 @@ module.exports = {
@@ -35,6 +35,7 @@ module.exports = {
offsite,
offsite,
archive: require('./archive'),
archive: require('./archive'),
alerts: require('./alerts'),
alerts: require('./alerts'),
+ browserWatch: require('./browser-watch'),
snapshot,
snapshot,
drill,
drill,
DrillError,
DrillError,
@@ -173,9 +173,9 @@ runTests([
@@ -173,9 +173,9 @@ runTests([
try {
try {
const script = path.join(dir, 'check.js');
const script = path.join(dir, 'check.js');
fs.writeFileSync(script, "const a = process.argv.slice(2); if (a[1] === 'down') { console.error('Error: Chrome not found (set CHROME_BIN)'); process.exit(2); } console.log(`ok site ${a.join(' ')}`); process.exit(a[1] === 'bad' ? 1 : 0);");
fs.writeFileSync(script, "const a = process.argv.slice(2); if (a[1] === 'down') { console.error('Error: Chrome not found (set CHROME_BIN)'); process.exit(2); } console.log(`ok site ${a.join(' ')}`); process.exit(a[1] === 'bad' ? 1 : 0);");
- assert.deepStrictEqual(await runBrowserCheck('wiki', { script, args: ['--no-nav'] }), { code: 0, lines: ['ok site --sites wiki --no-nav'], error: null });
+ assert.deepStrictEqual(await runBrowserCheck('wiki', { script, args: ['--no-nav'] }), { code: 0, lines: ['ok site --sites wiki --no-nav'], stdout: 'ok site --sites wiki --no-nav\n', error: null });
assert.strictEqual((await runBrowserCheck('bad', { script })).code, 1);
assert.strictEqual((await runBrowserCheck('bad', { script })).code, 1);
- assert.deepStrictEqual(await runBrowserCheck('down', { script }), { code: 2, lines: [], error: 'Error: Chrome not found (set CHROME_BIN)' });
+ assert.deepStrictEqual(await runBrowserCheck('down', { script }), { code: 2, lines: [], stdout: '', error: 'Error: Chrome not found (set CHROME_BIN)' });
fs.writeFileSync(script, 'setTimeout(() => {}, 60000);');
fs.writeFileSync(script, 'setTimeout(() => {}, 60000);');
const slow = await runBrowserCheck('x', { script, timeoutMs: 300 });
const slow = await runBrowserCheck('x', { script, timeoutMs: 300 });
assert.strictEqual(slow.code, 2);
assert.strictEqual(slow.code, 2);
new file mode 100644
new file mode 100644
@@ -0,0 +1,89 @@
@@ -0,0 +1,89 @@
+'use strict';
+/**
+ * The release UX check (lib/browser-watch.js, WS-H task 11): a site is checked in a browser when its release
+ * changes and once a day, as ovcheck; a failure is confirmed by a second run; a run that broke records
+ * nothing; the outcome reaches the textfile collector for OpenVibeBrowserCheckFailed.
+ */
+const assert = require('assert');
+const { spawn } = require('child_process');
+const { scenario, test, runTests } = require('./helpers');
+const { watch, failingChecks, DAY_MS } = require('../lib/browser-watch');
+const { runBrowserCheck } = require('../lib/browser-check-hook');
+
+const PROM = '/var/lib/prometheus/node-exporter/openvibe_browsercheck.prom';
+const pass = [{ base: 'https://openvibe.live', ok: true, summary: { checks: { status: { fail: 0 }, console: { fail: 0 } } } }];
+const fail = [{ base: 'https://openvibe.live', ok: false, summary: { checks: { status: { fail: 0 }, console: { fail: 2 }, overflow: { fail: 1 } } } }];
+
+function site({ release = 'abc1234def56' } = {}) {
+ const host = scenario();
+ host.inv.services.live.origin = 'https://openvibe.live';
+ host.put('/var/lib/prometheus/node-exporter/.keep', '');
+ host.release = release;
+ host.http.set('https://openvibe.live/release.json', () => ({ status: 200, body: { service: 'live', release: host.release } }));
+ host.runs = [];
+ host.answers = [];
+ host.check = async (s, opts) => { host.runs.push({ s, opts }); const a = host.answers.shift() || { code: 0, reports: pass }; return { code: a.code, stdout: a.reports ? JSON.stringify(a.reports) : '', error: a.error || null }; };
+ return host;
+}
+
+runTests([
+ test('a new release is checked as ovcheck with --json; the same release is skipped; a day later it is checked again', async () => {
+ const host = site();
+ const now = Date.parse('2026-09-26T14:00:00Z');
+ let r = await watch(host.ctx(), { check: host.check, now });
+ assert.deepStrictEqual(r.sites.map((x) => [x.site, x.release, x.action, x.ok]), [['live', 'abc1234def56', 'checked', true]]);
+ assert.deepStrictEqual(host.runs.map((x) => [x.s, x.opts.asUser, x.opts.home, x.opts.args]), [['live', 'ovcheck', '/var/lib/ovcheck', ['--json']]]);
+ assert.match(host.files.get(PROM).content, /^openvibe_browser_check_ok\{service="live",release="abc1234def56"\} 1$/m);
+ assert.match(host.files.get(PROM).content, /^openvibe_browser_check_failing_checks\{service="live"\} 0$/m);
+ r = await watch(host.ctx(), { check: host.check, now: now + 60000 });
+ assert.strictEqual(r.sites[0].action, 'skipped');
+ assert.strictEqual(host.runs.length, 1);
+ r = await watch(host.ctx(), { check: host.check, now: now + DAY_MS + 1 });
+ assert.strictEqual(r.sites[0].action, 'checked', 'daily even without a release');
+ host.release = 'fff0000aaa11';
+ r = await watch(host.ctx(), { check: host.check, now: now + DAY_MS + 60000 });
+ assert.deepStrictEqual([r.sites[0].action, r.sites[0].release], ['checked', 'fff0000aaa11']);
+ r = await watch(host.ctx(), { check: host.check, now: now + DAY_MS + 120000, force: true });
+ assert.strictEqual(r.sites[0].action, 'checked', '--force');
+ }),
+ test('a failure is confirmed by a second run before it counts; one flake then a pass is a pass', async () => {
+ let host = site();
+ host.answers.push({ code: 1, reports: fail }, { code: 1, reports: fail });
+ let r = await watch(host.ctx(), { check: host.check });
+ assert.strictEqual(host.runs.length, 2);
+ assert.deepStrictEqual([r.sites[0].ok, r.sites[0].failing], [false, [{ check: 'console', fail: 2 }, { check: 'overflow', fail: 1 }]]);
+ assert.match(host.files.get(PROM).content, /^openvibe_browser_check_ok\{service="live",release="abc1234def56"\} 0$/m);
+ assert.match(host.files.get(PROM).content, /^openvibe_browser_check_failing_checks\{service="live"\} 2$/m);
+ host = site();
+ host.answers.push({ code: 1, reports: fail }, { code: 0, reports: pass });
+ r = await watch(host.ctx(), { check: host.check });
+ assert.deepStrictEqual([host.runs.length, r.sites[0].ok], [2, true]);
+ }),
+ test('a run that broke (exit 2) records nothing and is retried by the next timer run', async () => {
+ const host = site();
+ host.answers.push({ code: 2, error: 'Chrome not found' });
+ let r = await watch(host.ctx(), { check: host.check });
+ assert.deepStrictEqual([r.sites[0].action, r.sites[0].reason, host.runs.length], ['error', 'Chrome not found', 1]);
+ assert.ok(!host.files.has('/var/lib/openvibe-host/browser-watch/live.json'));
+ r = await watch(host.ctx(), { check: host.check });
+ assert.strictEqual(r.sites[0].action, 'checked');
+ }),
+ test('bad ids and sites without an origin are errors, not checks; the CLI exits 1 on a failed check and 2 on an error', async () => {
+ const host = site();
+ const r = await watch(host.ctx(), { check: host.check, sites: ['Bad!', 'nosuchservice'], contracts: { services: { get: () => null } } });
+ assert.deepStrictEqual(r.sites.map((x) => x.action), ['error', 'error']);
+ assert.strictEqual(host.runs.length, 0);
+ const c = await scenario().cli('browser-watch', '--sites', 'Bad!');
+ assert.strictEqual(c.code, 2);
+ assert.match(c.out, /ERR {2}Bad!: not a service id/);
+ }),
+ test('failingChecks reads the reports; the hook runs as another user through runuser with HOME', async () => {
+ assert.deepStrictEqual(failingChecks([{ summary: { checks: { a: { fail: 0 }, b: { fail: 3 } } } }, { error: 'boom', summary: { checks: {} } }]), [{ check: 'b', fail: 3 }, { check: 'run', fail: 1 }]);
+ const seen = [];
+ const fake = (cmd, args) => { seen.push([cmd, ...args]); return spawn(process.execPath, ['-e', 'process.stdout.write("[]")']); };
+ const r = await runBrowserCheck('live', { args: ['--json'], asUser: 'ovcheck', home: '/var/lib/ovcheck', spawnFn: fake });
+ assert.deepStrictEqual([r.code, r.stdout], [0, '[]']);
+ assert.deepStrictEqual(seen[0].slice(0, 6), ['runuser', '-u', 'ovcheck', '--', 'env', 'HOME=/var/lib/ovcheck']);
+ assert.ok(seen[0].includes('--sites') && seen[0].includes('live') && seen[0].includes('--json'));
+ }),
+]);