openvibers/openvibe.host Commits

The release UX check (roadmap WS-H task 11): ovhost browser-watch --sites live (openvibe-browsercheck.timer, every 5 minutes) checks each watched site in Chrome after every release (read from /release.json) and once a day. Chrome runs as the unprivileged ovcheck account through runuser, and a failure is confirmed by a second run. The outcome goes to openvibe_browsercheck.prom: OpenVibeBrowserCheckFailed (page) goes through ovhost alerts relay to the operator, and OpenVibeBrowserCheckMissed notices when no check finished for 26 hours. A run that broke records nothing and is retried. The browser-check hook gains asUser/home and returns stdout. docs/operator-alerts.md; test/browser-watch.test.js

OP OpenVibers committed 26 Sept 2026 · 9d4c552dbe · 10 files, +311 −8


Changes

ddiff --git a/deploy/prometheus/openvibe-rules.yml b/deploy/prometheus/openvibe-rules.yml
iindex 40d5f5d..803941e 100644
---- a/deploy/prometheus/openvibe-rules.yml
++++ b/deploy/prometheus/openvibe-rules.yml
@@@ -79,6 +79,20 @@ groups:
labels: { severity: ticket }
annotations:
summary: "No passing Tools job proof for more than 13 hours (two runs)"
++ # The release UX check (ovhost browser-watch, openvibe-browsercheck.timer; WS-H task 11): each watched site
++ # in a real browser after every release and daily, as ovcheck; a failure is confirmed by a second run.
++ - alert: OpenVibeBrowserCheckFailed
++ expr: openvibe_browser_check_ok == 0
++ for: 1m
++ labels: { severity: page }
++ annotations:
++ summary: "{{ $labels.service }}: the browser check of release {{ $labels.release }} failed ({{ with printf \"openvibe_browser_check_failing_checks{service='%s'}\" $labels.service | query }}{{ . | first | value }}{{ end }} failing check(s)); run: sudo ovhost browser-watch --sites {{ $labels.service }} --force"
++ - alert: OpenVibeBrowserCheckMissed
++ expr: time() - openvibe_browser_check_last_run_timestamp_seconds > 26 * 3600
++ for: 30m
++ labels: { severity: ticket }
++ annotations:
++ summary: "{{ $labels.service }}: no browser check has run to the end for more than 26 hours"
# The relay itself (openvibe-alerts.timer; WS-H task 11). It cannot page while it is broken; it pages
# this as soon as a run gets through again, so an outage of the pager is never silent afterwards.
- alert: OpenVibeAlertRelayFailing
ddiff --git a/deploy/systemd/openvibe-browsercheck.service b/deploy/systemd/openvibe-browsercheck.service
nnew file mode 100644
iindex 0000000..1714c63
---- /dev/null
++++ b/deploy/systemd/openvibe-browsercheck.service
@@@ -0,0 +1,18 @@
++[Unit]
++Description=OpenVibe release UX check: each watched site in a real browser after every release (WS-H task 11)
++Documentation=https://github.com/OpenVibers/OpenVibe.Host/blob/main/docs/operator-alerts.md
++After=network-online.target
++Wants=network-online.target
++
++[Service]
++Type=oneshot
++# Root reads the state and writes the textfile metrics; Chrome runs as the unprivileged ovcheck account
++# (runuser), never as root. --sites names the watched services (inventory origin or manifest publicOrigin).
++User=root
++ExecStart=/usr/local/bin/ovhost browser-watch --sites live
++Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
++Environment=OVHOST_INVENTORY=/etc/openvibe/host.json
++Nice=10
++TimeoutStartSec=30min
++PrivateTmp=true
++ProtectSystem=full
ddiff --git a/deploy/systemd/openvibe-browsercheck.timer b/deploy/systemd/openvibe-browsercheck.timer
nnew file mode 100644
iindex 0000000..6b81cf5
---- /dev/null
++++ b/deploy/systemd/openvibe-browsercheck.timer
@@@ -0,0 +1,10 @@
++[Unit]
++Description=Check each watched OpenVibe site in a browser after a release (every five minutes; WS-H task 11)
++
++[Timer]
++OnBootSec=5min
++OnUnitActiveSec=5min
++AccuracySec=30s
++
++[Install]
++WantedBy=timers.target
ddiff --git a/docs/operator-alerts.md b/docs/operator-alerts.md
iindex 141d798..d8a0a6a 100644
---- a/docs/operator-alerts.md
++++ b/docs/operator-alerts.md
@@@ -31,11 +31,32 @@ Prometheus (127.0.0.1:9090) ──GET /api/v1/alerts──▶ ovhost alerts re
- **Relay failures**: `openvibe_alert_relay_last_run_ok` and `openvibe_alert_relay_last_success_timestamp_seconds`
record them, and `OpenVibeAlertRelayFailing` pages as soon as delivery works again.
++## The release UX check
++
++`ovhost browser-watch --sites live` (openvibe-browsercheck.timer, every 5 minutes) reads each watched site's
++`/release.json`. For a release it has not checked yet, or once a day, it runs
++[scripts/browser-check.js](../scripts/browser-check.js) in Chrome as the unprivileged `ovcheck` account (never
++root): routes × widths, console errors, overflow, duplicate scripts, no-JS content, axe, navigation growth.
++
++- **Confirmation**: a failure is re-run once before it counts.
++- **Outcome**: the result is kept in `/var/lib/openvibe-host/browser-watch/<site>.json` and written to
++ `openvibe_browsercheck.prom`.
++- **Paging**: `OpenVibeBrowserCheckFailed` (severity page) pages through the relay, naming the release and the
++ number of failing checks.
++- **Runs that never finish**: a run that broke (exit 2, e.g. Chrome missing) records nothing and is retried
++ on the next tick, and `OpenVibeBrowserCheckMissed` notices a site with no finished check for 26 hours.
++- **Cost**: a check of openvibe.live takes about a minute on the host.
++- **Setup**: Chrome comes from Google's apt repository (`google-chrome-stable`, updated with the system), and
++ `ovcheck` is a system account whose home is `/var/lib/ovcheck`.
++
++To watch more sites, add them to `--sites` in the unit.
++
## Commands
```bash
sudo ovhost alerts relay --dry-run # what is firing, nothing sent
sudo ovhost alerts relay # deliver now
++sudo ovhost browser-watch --sites live --force # check the current release now
systemctl list-timers openvibe-alerts.timer
```
@@@ -43,7 +64,13 @@ systemctl list-timers openvibe-alerts.timer
```bash
sudo cp /usr/local/lib/openvibe-host/deploy/systemd/openvibe-alerts.{service,timer} /etc/systemd/system/
--sudo systemctl daemon-reload && sudo systemctl enable --now openvibe-alerts.timer
++sudo cp /usr/local/lib/openvibe-host/deploy/systemd/openvibe-browsercheck.{service,timer} /etc/systemd/system/
++sudo systemctl daemon-reload && sudo systemctl enable --now openvibe-alerts.timer openvibe-browsercheck.timer
++# Chrome and the check account (once):
++curl -fsSL https://dl.google.com/linux/linux_signing_key.pub | sudo gpg --dearmor -o /usr/share/keyrings/google-chrome.gpg
++echo "deb [arch=amd64 signed-by=/usr/share/keyrings/google-chrome.gpg] https://dl.google.com/linux/chrome/deb/ stable main" | sudo tee /etc/apt/sources.list.d/google-chrome.list
++sudo apt-get update && sudo apt-get install -y google-chrome-stable
++sudo useradd --system --home-dir /var/lib/ovcheck --create-home --shell /usr/sbin/nologin ovcheck
sudo cp /usr/local/lib/openvibe-host/deploy/prometheus/openvibe-rules.yml /etc/prometheus/rules/openvibe.yml
sudo promtool check rules /etc/prometheus/rules/openvibe.yml && sudo systemctl reload prometheus
```
ddiff --git a/lib/browser-check-hook.js b/lib/browser-check-hook.js
iindex 7d2a068..53a5639 100644
---- a/lib/browser-check-hook.js
++++ b/lib/browser-check-hook.js
@@@ -10,10 +10,17 @@ const path = require('path');
const SCRIPT = path.join(__dirname, '..', 'scripts', 'browser-check.js');
--/** Answers { code, lines, error }: code 0 pass, 1 problems found, 2 (or a signal) the check did not run. */
--function runBrowserCheck(service, { timeoutMs = 15 * 60 * 1000, script = SCRIPT, args = [] } = {}) {
++/**
++ * Answers { code, lines, stdout, error }: code 0 pass, 1 problems found, 2 (or a signal) the check did not run.
++ * asUser runs it as that account through runuser (Chrome never runs as root; ovhost browser-watch uses ovcheck),
++ * with HOME set to home.
++ */
++function runBrowserCheck(service, { timeoutMs = 15 * 60 * 1000, script = SCRIPT, args = [], asUser = null, home = null, spawnFn = spawn } = {}) {
return new Promise((resolve) => {
-- const child = spawn(process.execPath, [script, '--sites', service, ...args], { stdio: ['ignore', 'pipe', 'pipe'] });
++ const argv = [script, '--sites', service, ...args];
++ const child = asUser
++ ? spawnFn('runuser', ['-u', asUser, '--', 'env', `HOME=${home || '/tmp'}`, process.execPath, ...argv], { stdio: ['ignore', 'pipe', 'pipe'] })
++ : spawnFn(process.execPath, argv, { stdio: ['ignore', 'pipe', 'pipe'] });
let out = '', err = '';
child.stdout.on('data', (c) => { out += c; });
child.stderr.on('data', (c) => { err += c; });
@@@ -24,7 +31,7 @@ function runBrowserCheck(service, { timeoutMs = 15 * 60 * 1000, script = SCRIPT,
const lines = out.split('\n').map((l) => l.trim()).filter(Boolean);
const errLines = err.split('\n').map((l) => l.trim()).filter(Boolean);
const c = code == null ? 2 : code;
-- resolve({ code: c, lines, error: c >= 2 ? (errLines.find((l) => /Error|not found|timed out/.test(l)) || errLines[errLines.length - 1] || signal || `exit ${c}`) : null });
++ resolve({ code: c, lines, stdout: out, error: c >= 2 ? (errLines.find((l) => /Error|not found|timed out/.test(l)) || errLines[errLines.length - 1] || signal || `exit ${c}`) : null });
});
});
}
ddiff --git a/lib/browser-watch.js b/lib/browser-watch.js
nnew file mode 100644
iindex 0000000..e162e98
---- /dev/null
++++ b/lib/browser-watch.js
@@@ -0,0 +1,124 @@
++'use strict';
++/**
++ * `ovhost browser-watch` (roadmap WS-H task 11, "release UX check"): each watched site is checked in a real
++ * browser (scripts/browser-check.js through lib/browser-check-hook.js) after every release, and once a day
++ * without one, and a failure pages the operator (OpenVibeBrowserCheckFailed → ovhost alerts relay → Network).
++ *
++ * ovhost browser-watch [--sites live,network] [--force] (openvibe-browsercheck.timer: every 5 minutes)
++ *
++ * Per site: GET <origin>/release.json. A release not checked yet, or a last check older than 24 hours, runs the
++ * check as the unprivileged `ovcheck` account (Chrome never runs as root). A failing check runs once more before
++ * it counts (a single network blip is not a regression). The outcome is kept in
++ * <stateDir>/browser-watch/<site>.json, and every site's latest outcome is written to the textfile collector
++ * (openvibe_browsercheck.prom):
++ *
++ * openvibe_browser_check_ok{service,release} 1 passed, 0 failed
++ * openvibe_browser_check_failing_checks{service} checks with failures (status, console, overflow, axe…)
++ * openvibe_browser_check_last_run_timestamp_seconds{service} when the last check ran to the end
++ *
++ * A check that could not run (Chrome missing, the harness broke: exit 2) is not recorded as a pass or a fail;
++ * the next timer run tries again, and OpenVibeBrowserCheckMissed notices when none has run for a day.
++ * The origin comes from the inventory entry's `origin`, else the service manifest's publicOrigin.
++ */
++const path = require('path');
++const { runBrowserCheck } = require('./browser-check-hook');
++
++const DAY_MS = 24 * 3600 * 1000;
++const TEXTFILE_DIR = '/var/lib/prometheus/node-exporter';
++const SITE_RE = /^[a-z][a-z0-9-]{1,39}$/;
++const RELEASE_RE = /^[0-9A-Za-z._+-]{1,64}$/;
++
++function originOf(inv, id, contracts) {
++ const entry = inv.services && inv.services[id];
++ if (entry && entry.origin) return String(entry.origin).replace(/\/+$/, '');
++ let lib = contracts;
++ if (!lib) { try { lib = require('openvibe-contracts'); } catch { return null; } }
++ const m = lib.services && typeof lib.services.get === 'function' ? lib.services.get(id) : null;
++ return m && m.publicOrigin ? String(m.publicOrigin).replace(/\/+$/, '') : null;
++}
++
++async function currentRelease(exec, origin) {
++ const r = await exec.request(`${origin}/release.json`, { timeoutMs: 8000 });
++ if (r.status !== 200) return null;
++ try {
++ const m = JSON.parse(r.body);
++ const rel = String(m.release || m.commit || '').slice(0, 12);
++ return RELEASE_RE.test(rel) ? rel : null;
++ } catch { return null; }
++}
++
++/** The failing checks of a --json report list: [{ check, fail }]. */
++function failingChecks(reports) {
++ const out = [];
++ for (const r of reports || []) {
++ const checks = (r.summary && r.summary.checks) || {};
++ for (const [k, c] of Object.entries(checks)) if (c && c.fail) out.push({ check: k, fail: c.fail });
++ if (r.error) out.push({ check: 'run', fail: 1 });
++ }
++ return out;
++}
++
++const esc = (v) => String(v).replace(/\\/g, '\\\\').replace(/"/g, '\\"');
++
++async function writeMetrics(exec, inv) {
++ const dir = inv.textfileDir || TEXTFILE_DIR;
++ if (!(await exec.readdir(dir))) return false;
++ const stateDir = path.join(inv.stateDir, 'browser-watch');
++ const rows = [];
++ for (const e of (await exec.readdir(stateDir)) || []) {
++ if (!e.name.endsWith('.json')) continue;
++ try { const s = JSON.parse((await exec.readFile(path.join(stateDir, e.name))) || 'null'); if (s && SITE_RE.test(s.site)) rows.push(s); } catch { /* skip */ }
++ }
++ rows.sort((a, b) => a.site.localeCompare(b.site));
++ const lines = [
++ '# HELP openvibe_browser_check_ok 1 when the last browser check of the site passed (label: the release it checked).',
++ '# TYPE openvibe_browser_check_ok gauge',
++ ...rows.map((s) => `openvibe_browser_check_ok{service="${esc(s.site)}",release="${esc(s.release || 'unknown')}"} ${s.ok ? 1 : 0}`),
++ '# HELP openvibe_browser_check_failing_checks Checks with failures in the last browser check of the site.',
++ '# TYPE openvibe_browser_check_failing_checks gauge',
++ ...rows.map((s) => `openvibe_browser_check_failing_checks{service="${esc(s.site)}"} ${(s.failing || []).length}`),
++ '# HELP openvibe_browser_check_last_run_timestamp_seconds When the last browser check of the site ran to the end.',
++ '# TYPE openvibe_browser_check_last_run_timestamp_seconds gauge',
++ ...rows.map((s) => `openvibe_browser_check_last_run_timestamp_seconds{service="${esc(s.site)}"} ${Math.floor(Date.parse(s.checkedAt) / 1000)}`),
++ ];
++ await exec.writeFile(path.join(dir, 'openvibe_browsercheck.prom'), `${lines.join('\n')}\n`, { mode: 0o644 });
++ return true;
++}
++
++/**
++ * One pass over the sites. Answers { sites: [{ site, release, action: 'checked'|'skipped'|'error', ok?, failing?, reason? }] }.
++ * opts: sites (ids), force, asUser (default ovcheck), home, now, check (the runBrowserCheck to use; tests).
++ */
++async function watch(ctx, { sites = ['live'], force = false, asUser = 'ovcheck', home = '/var/lib/ovcheck', now = Date.now(), check = runBrowserCheck, contracts = null } = {}) {
++ const { exec, inv } = ctx;
++ const stateDir = path.join(inv.stateDir, 'browser-watch');
++ await exec.mkdir(stateDir, { mode: 0o750 });
++ const out = [];
++ for (const site of sites) {
++ if (!SITE_RE.test(site)) { out.push({ site, action: 'error', reason: 'not a service id' }); continue; }
++ const origin = originOf(inv, site, contracts);
++ if (!origin) { out.push({ site, action: 'error', reason: 'no origin (inventory origin or manifest publicOrigin)' }); continue; }
++ const release = await currentRelease(exec, origin);
++ const file = path.join(stateDir, `${site}.json`);
++ let prev = null;
++ try { prev = JSON.parse((await exec.readFile(file)) || 'null'); } catch { prev = null; }
++ const fresh = prev && prev.release === release && now - Date.parse(prev.checkedAt) < DAY_MS;
++ if (fresh && !force) { out.push({ site, release, action: 'skipped', ok: prev.ok, reason: 'this release was checked' }); continue; }
++ let r = null;
++ let reports = null;
++ for (let attempt = 1; attempt <= 2; attempt++) {
++ r = await check(site, { args: ['--json'], asUser, home });
++ try { reports = JSON.parse(r.stdout || 'null'); } catch { reports = null; }
++ if (r.code === 0 || r.code >= 2 || !reports) break; // a pass, or a run that broke: no second attempt
++ }
++ if (r.code >= 2 || !Array.isArray(reports)) { out.push({ site, release, action: 'error', reason: r.error || `the check did not run (exit ${r.code})` }); continue; }
++ const failing = failingChecks(reports);
++ const state = { site, origin, release, ok: r.code === 0, failing, checkedAt: new Date(now).toISOString() };
++ await exec.writeFile(file, `${JSON.stringify(state, null, 2)}\n`, { mode: 0o640 });
++ out.push({ site, release, action: 'checked', ok: state.ok, failing });
++ }
++ try { await writeMetrics(exec, inv); } catch { /* metrics never fail a run */ }
++ return { sites: out };
++}
++
++module.exports = { watch, failingChecks, writeMetrics, originOf, currentRelease, DAY_MS };
ddiff --git a/lib/cli.js b/lib/cli.js
iindex 71a47ff..66b4f7d 100644
---- a/lib/cli.js
++++ b/lib/cli.js
@@@ -62,6 +62,10 @@ Usage: ovhost <command> [args] [--inventory <file>] [--json]
declared by the inventory) to <prefix>-archive/; never deletes them
archive list the archives off-host
archive restore <stamp> <path> --out <dir> one archived file back, checked against the manifest
++ browser-watch [--sites live,...] [--force]
++ check each site in a real browser after every release (and
++ daily) as ovcheck; metrics for OpenVibeBrowserCheckFailed
++ (openvibe-browsercheck.timer, every 5 minutes)
alerts relay [--prometheus <url>] [--dry-run]
the alerts firing in Prometheus, sent to Network, which pages
the operator (openvibe-alerts.timer, every 2 minutes)
@@@ -83,7 +87,7 @@ drill: 0 passed · 1 precondition (not root, port in use, no backup, unsupported
backup --all / offsite: 0 everything succeeded · 1 precondition (config, key, lock) · 2 something failed
`;
--const FLAGS_WITH_VALUE = new Set(['prometheus', 'inventory', 'to', 'limit', 'warn-days', 'variant', 'manifest', 'out', 'ready-timeout', 'wildcard-cert', 'backup', 'run', 'keep-daily', 'keep-weekly', 'from-host', 'backup-env', 'release', 'commit', 'origin', 'events-env', 'note']);
++const FLAGS_WITH_VALUE = new Set(['prometheus', 'sites', 'inventory', 'to', 'limit', 'warn-days', 'variant', 'manifest', 'out', 'ready-timeout', 'wildcard-cert', 'backup', 'run', 'keep-daily', 'keep-weekly', 'from-host', 'backup-env', 'release', 'commit', 'origin', 'events-env', 'note']);
function parseArgs(argv) {
const pos = [];
@@@ -370,6 +374,15 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc
else out(`OFFSITE ${o.ok ? 'ok' : 'FAILED'}: ${o.uploaded.length} file(s) to s3://${o.bucket}/${o.location}${o.failures.length ? `; failures: ${o.failures.map((f) => `${f.service ? `${f.service}/` : ''}${f.file}: ${f.error}`).join('; ')}` : ''}`);
return o.ok ? 0 : 2;
}
++ case 'browser-watch': {
++ // WS-H task 11: the release UX check. A failure pages through the alerts relay.
++ const sites = String(opts.sites || 'live').split(',').map((x) => x.trim()).filter(Boolean);
++ const r = await lib.browserWatch.watch(ctx, { sites, force: !!opts.force });
++ if (json) emit(r);
++ else for (const x of r.sites) out(x.action === 'checked' ? `${x.ok ? 'ok ' : 'FAIL'} ${x.site} ${x.release || '?'}${x.failing && x.failing.length ? `; failing: ${x.failing.map((f) => `${f.check} ${f.fail}`).join(', ')}` : ''}`
++ : x.action === 'skipped' ? `skip ${x.site} ${x.release || '?'} (${x.reason}; last ${x.ok ? 'passed' : 'failed'})` : `ERR ${x.site}: ${x.reason}`);
++ return r.sites.some((x) => x.action === 'error') ? 2 : r.sites.some((x) => x.action === 'checked' && !x.ok) ? 1 : 0;
++ }
case 'alerts': {
// WS-H task 11: deliver the firing Prometheus alerts to the operator through Network.
if (pos[1] !== 'relay') throw new UsageError('usage: ovhost alerts relay [--prometheus <url>] [--dry-run] [--events-env <file>]');
ddiff --git a/lib/index.js b/lib/index.js
iindex 08a2a93..de756d4 100644
---- a/lib/index.js
++++ b/lib/index.js
@@@ -35,6 +35,7 @@ module.exports = {
offsite,
archive: require('./archive'),
alerts: require('./alerts'),
++ browserWatch: require('./browser-watch'),
snapshot,
drill,
DrillError,
ddiff --git a/test/browser-check.test.js b/test/browser-check.test.js
iindex b084781..8f39bb5 100644
---- a/test/browser-check.test.js
++++ b/test/browser-check.test.js
@@@ -173,9 +173,9 @@ runTests([
try {
const script = path.join(dir, 'check.js');
fs.writeFileSync(script, "const a = process.argv.slice(2); if (a[1] === 'down') { console.error('Error: Chrome not found (set CHROME_BIN)'); process.exit(2); } console.log(`ok site ${a.join(' ')}`); process.exit(a[1] === 'bad' ? 1 : 0);");
-- assert.deepStrictEqual(await runBrowserCheck('wiki', { script, args: ['--no-nav'] }), { code: 0, lines: ['ok site --sites wiki --no-nav'], error: null });
++ assert.deepStrictEqual(await runBrowserCheck('wiki', { script, args: ['--no-nav'] }), { code: 0, lines: ['ok site --sites wiki --no-nav'], stdout: 'ok site --sites wiki --no-nav\n', error: null });
assert.strictEqual((await runBrowserCheck('bad', { script })).code, 1);
-- assert.deepStrictEqual(await runBrowserCheck('down', { script }), { code: 2, lines: [], error: 'Error: Chrome not found (set CHROME_BIN)' });
++ assert.deepStrictEqual(await runBrowserCheck('down', { script }), { code: 2, lines: [], stdout: '', error: 'Error: Chrome not found (set CHROME_BIN)' });
fs.writeFileSync(script, 'setTimeout(() => {}, 60000);');
const slow = await runBrowserCheck('x', { script, timeoutMs: 300 });
assert.strictEqual(slow.code, 2);
ddiff --git a/test/browser-watch.test.js b/test/browser-watch.test.js
nnew file mode 100644
iindex 0000000..dde1ee8
---- /dev/null
++++ b/test/browser-watch.test.js
@@@ -0,0 +1,89 @@
++'use strict';
++/**
++ * The release UX check (lib/browser-watch.js, WS-H task 11): a site is checked in a browser when its release
++ * changes and once a day, as ovcheck; a failure is confirmed by a second run; a run that broke records
++ * nothing; the outcome reaches the textfile collector for OpenVibeBrowserCheckFailed.
++ */
++const assert = require('assert');
++const { spawn } = require('child_process');
++const { scenario, test, runTests } = require('./helpers');
++const { watch, failingChecks, DAY_MS } = require('../lib/browser-watch');
++const { runBrowserCheck } = require('../lib/browser-check-hook');
++
++const PROM = '/var/lib/prometheus/node-exporter/openvibe_browsercheck.prom';
++const pass = [{ base: 'https://openvibe.live', ok: true, summary: { checks: { status: { fail: 0 }, console: { fail: 0 } } } }];
++const fail = [{ base: 'https://openvibe.live', ok: false, summary: { checks: { status: { fail: 0 }, console: { fail: 2 }, overflow: { fail: 1 } } } }];
++
++function site({ release = 'abc1234def56' } = {}) {
++ const host = scenario();
++ host.inv.services.live.origin = 'https://openvibe.live';
++ host.put('/var/lib/prometheus/node-exporter/.keep', '');
++ host.release = release;
++ host.http.set('https://openvibe.live/release.json', () => ({ status: 200, body: { service: 'live', release: host.release } }));
++ host.runs = [];
++ host.answers = [];
++ host.check = async (s, opts) => { host.runs.push({ s, opts }); const a = host.answers.shift() || { code: 0, reports: pass }; return { code: a.code, stdout: a.reports ? JSON.stringify(a.reports) : '', error: a.error || null }; };
++ return host;
++}
++
++runTests([
++ test('a new release is checked as ovcheck with --json; the same release is skipped; a day later it is checked again', async () => {
++ const host = site();
++ const now = Date.parse('2026-09-26T14:00:00Z');
++ let r = await watch(host.ctx(), { check: host.check, now });
++ assert.deepStrictEqual(r.sites.map((x) => [x.site, x.release, x.action, x.ok]), [['live', 'abc1234def56', 'checked', true]]);
++ assert.deepStrictEqual(host.runs.map((x) => [x.s, x.opts.asUser, x.opts.home, x.opts.args]), [['live', 'ovcheck', '/var/lib/ovcheck', ['--json']]]);
++ assert.match(host.files.get(PROM).content, /^openvibe_browser_check_ok\{service="live",release="abc1234def56"\} 1$/m);
++ assert.match(host.files.get(PROM).content, /^openvibe_browser_check_failing_checks\{service="live"\} 0$/m);
++ r = await watch(host.ctx(), { check: host.check, now: now + 60000 });
++ assert.strictEqual(r.sites[0].action, 'skipped');
++ assert.strictEqual(host.runs.length, 1);
++ r = await watch(host.ctx(), { check: host.check, now: now + DAY_MS + 1 });
++ assert.strictEqual(r.sites[0].action, 'checked', 'daily even without a release');
++ host.release = 'fff0000aaa11';
++ r = await watch(host.ctx(), { check: host.check, now: now + DAY_MS + 60000 });
++ assert.deepStrictEqual([r.sites[0].action, r.sites[0].release], ['checked', 'fff0000aaa11']);
++ r = await watch(host.ctx(), { check: host.check, now: now + DAY_MS + 120000, force: true });
++ assert.strictEqual(r.sites[0].action, 'checked', '--force');
++ }),
++ test('a failure is confirmed by a second run before it counts; one flake then a pass is a pass', async () => {
++ let host = site();
++ host.answers.push({ code: 1, reports: fail }, { code: 1, reports: fail });
++ let r = await watch(host.ctx(), { check: host.check });
++ assert.strictEqual(host.runs.length, 2);
++ assert.deepStrictEqual([r.sites[0].ok, r.sites[0].failing], [false, [{ check: 'console', fail: 2 }, { check: 'overflow', fail: 1 }]]);
++ assert.match(host.files.get(PROM).content, /^openvibe_browser_check_ok\{service="live",release="abc1234def56"\} 0$/m);
++ assert.match(host.files.get(PROM).content, /^openvibe_browser_check_failing_checks\{service="live"\} 2$/m);
++ host = site();
++ host.answers.push({ code: 1, reports: fail }, { code: 0, reports: pass });
++ r = await watch(host.ctx(), { check: host.check });
++ assert.deepStrictEqual([host.runs.length, r.sites[0].ok], [2, true]);
++ }),
++ test('a run that broke (exit 2) records nothing and is retried by the next timer run', async () => {
++ const host = site();
++ host.answers.push({ code: 2, error: 'Chrome not found' });
++ let r = await watch(host.ctx(), { check: host.check });
++ assert.deepStrictEqual([r.sites[0].action, r.sites[0].reason, host.runs.length], ['error', 'Chrome not found', 1]);
++ assert.ok(!host.files.has('/var/lib/openvibe-host/browser-watch/live.json'));
++ r = await watch(host.ctx(), { check: host.check });
++ assert.strictEqual(r.sites[0].action, 'checked');
++ }),
++ test('bad ids and sites without an origin are errors, not checks; the CLI exits 1 on a failed check and 2 on an error', async () => {
++ const host = site();
++ const r = await watch(host.ctx(), { check: host.check, sites: ['Bad!', 'nosuchservice'], contracts: { services: { get: () => null } } });
++ assert.deepStrictEqual(r.sites.map((x) => x.action), ['error', 'error']);
++ assert.strictEqual(host.runs.length, 0);
++ const c = await scenario().cli('browser-watch', '--sites', 'Bad!');
++ assert.strictEqual(c.code, 2);
++ assert.match(c.out, /ERR {2}Bad!: not a service id/);
++ }),
++ test('failingChecks reads the reports; the hook runs as another user through runuser with HOME', async () => {
++ assert.deepStrictEqual(failingChecks([{ summary: { checks: { a: { fail: 0 }, b: { fail: 3 } } } }, { error: 'boom', summary: { checks: {} } }]), [{ check: 'b', fail: 3 }, { check: 'run', fail: 1 }]);
++ const seen = [];
++ const fake = (cmd, args) => { seen.push([cmd, ...args]); return spawn(process.execPath, ['-e', 'process.stdout.write("[]")']); };
++ const r = await runBrowserCheck('live', { args: ['--json'], asUser: 'ovcheck', home: '/var/lib/ovcheck', spawnFn: fake });
++ assert.deepStrictEqual([r.code, r.stdout], [0, '[]']);
++ assert.deepStrictEqual(seen[0].slice(0, 6), ['runuser', '-u', 'ovcheck', '--', 'env', 'HOME=/var/lib/ovcheck']);
++ assert.ok(seen[0].includes('--sites') && seen[0].includes('live') && seen[0].includes('--json'));
++ }),
++]);