openvibers/openvibe.host Commits

Readiness probe for every service, with not-ready and missed-run alerts (#289)

OP OpenVibers committed 7 days ago · cc8f76e9b6 · 8 files, +492 −4

Games answered /api/ready 503 for eight hours on 2026-11-05 while its process
and /metrics stayed up, so TargetDown never fired. deploy/scripts/ready-health.sh
reads the host inventory, GETs every service's ready.url with its ready.headers
and a 5 s timeout, and writes openvibe_service_ready{service} 1/0 plus
openvibe_service_ready_last_run_timestamp to node_exporter's textfile collector
every 5 minutes (openvibe-readyhealth.timer).

OpenVibeServiceNotReady pages after 15 minutes of 0. OpenVibeServiceReadyMissed
pages when no run has completed for 20 minutes, so a broken probe cannot look
like a healthy fleet. docs/ready-health.md covers the metric, exit codes and the
manual rule install; test/ready-health.test.js runs the script against local
HTTP servers.

Changes

ddiff --git a/README.md b/README.md
iindex 699a69d..b9bd590 100644
---- a/README.md
++++ b/README.md
@@@ -297,7 +297,7 @@ Two things deploy from this repository:
    `cd /usr/local/lib/openvibe-host && sudo git pull --ff-only && sudo npm ci --omit=dev --no-audit --no-fund`,
    then `ovhost --version` and `sudo ovhost capabilities` (ovhost does not deploy itself; an update is
    undone by checking out the previous commit there). Its timers (`openvibe-backup`, `openvibe-alerts`, `openvibe-browsercheck`,
--  `openvibe-authhealth`, `openvibe-devpath`, `openvibe-toolsjob`) are unit files in [deploy/systemd/](deploy/systemd/).
++  `openvibe-authhealth`, `openvibe-readyhealth`, `openvibe-devpath`, `openvibe-toolsjob`) are unit files in [deploy/systemd/](deploy/systemd/).
  - **Boot order.** Three drop-ins in [deploy/systemd/](deploy/systemd/) order every `openvibe-*` and `openre-*`
    service after PostgreSQL, PgBouncer and Valkey (systemd prefix drop-ins, `openvibe-.service.d/`), and PgBouncer
    after PostgreSQL. Without them a reboot starts services against a PgBouncer that caches PostgreSQL's refused
ddiff --git a/deploy/prometheus/openvibe-rules.yml b/deploy/prometheus/openvibe-rules.yml
iindex 7536bf8..c1d4494 100644
---- a/deploy/prometheus/openvibe-rules.yml
++++ b/deploy/prometheus/openvibe-rules.yml
@@@ -12,6 +12,21 @@ groups:
          labels: { severity: page }
          annotations:
            summary: "{{ $labels.process }} ({{ $labels.instance }}) has not answered /metrics for 5 minutes"
++      # Readiness of every service with a ready URL (deploy/scripts/ready-health.sh,
++      # openvibe-readyhealth.timer; docs/ready-health.md). Games answered /api/ready 503 for 8 hours on
++      # 2026-11-05 while its process and /metrics were up, so TargetDown never fired.
++      - alert: OpenVibeServiceNotReady
++        expr: openvibe_service_ready == 0
++        for: 15m
++        labels: { severity: page }
++        annotations:
++          summary: "{{ $labels.service }} has not been ready for 15 minutes (read its /api/ready; sudo ovhost status {{ $labels.service }})"
++      - alert: OpenVibeServiceReadyMissed
++        expr: time() - openvibe_service_ready_last_run_timestamp > 20 * 60 or absent(openvibe_service_ready_last_run_timestamp)
++        for: 5m
++        labels: { severity: page }
++        annotations:
++          summary: "The readiness check has not completed a run for more than 20 minutes (openvibe-readyhealth.timer; check systemctl status openvibe-readyhealth.service)"
        - alert: HighErrorRate
          expr: |
            sum by (service) (rate(http_requests_total{status=~"5.."}[10m]))
ddiff --git a/deploy/scripts/ready-health.sh b/deploy/scripts/ready-health.sh
nnew file mode 100755
iindex 0000000..dfa11b3
---- /dev/null
++++ b/deploy/scripts/ready-health.sh
@@@ -0,0 +1,151 @@
++#!/usr/bin/env bash
++# Readiness probe for every service with a ready URL (docs/ready-health.md, openvibe-readyhealth.timer).
++#
++#   deploy/scripts/ready-health.sh                 every service in the host inventory
++#   OVHOST_INVENTORY=<file>                        the inventory (default /etc/openvibe/host.json)
++#   TEXTFILE_DIR=<dir>                             node_exporter's textfile collector (tests)
++#
++# For each service in the inventory with a ready.url: GET it with the ready.headers and a 5 s timeout.
++# One textfile line per service, written to <TEXTFILE_DIR>/openvibe_service_ready.prom:
++#
++#   openvibe_service_ready{service="<id>"} 1   the ready URL answered 2xx: the service is ready
++#   openvibe_service_ready{service="<id>"} 0   any other answer (a 500, a 503, a redirect) or no answer
++#   openvibe_service_ready_last_run_timestamp <seconds>   when the run finished
++#
++# A service with no ready URL gets no line: this check does not judge what the inventory does not declare.
++# The file is written atomically (a temp file beside it, then rename), so Prometheus never reads half of it.
++# Exit: 0 every service is ready · 1 one is not · 2 the inventory could not be read or named no service with
++# a ready URL (the previous file is kept: an inventory that cannot be checked is not green).
++set -uo pipefail
++
++INVENTORY=${OVHOST_INVENTORY:-/etc/openvibe/host.json}
++DIR=${TEXTFILE_DIR:-/var/lib/prometheus/node-exporter}
++OUT=$DIR/openvibe_service_ready.prom
++
++usage() { sed -n '2,18p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; }
++
++while [ $# -gt 0 ]; do
++    case "$1" in
++        -h|--help) usage; exit 0 ;;
++        *) echo "ready-health: unknown argument $1" >&2; exit 2 ;;
++    esac
++done
++
++command -v curl >/dev/null || { echo "ready-health: curl is needed" >&2; exit 2; }
++command -v node >/dev/null || { echo "ready-health: node is needed" >&2; exit 2; }
++
++TMP=$(mktemp -d) || exit 2
++trap 'rm -rf "$TMP"' EXIT
++umask 077
++
++# The probes: id<TAB>curl-config, one line per service with a ready URL, sorted by id. Node validates the
++# service id (a Prometheus label value), the URL and the headers (a header name and a single-line value),
++# and quotes them into a curl config so a header value can hold spaces, colons and quotes. The inventory
++# is root-owned: this reads it, and writes nothing back.
++if ! node - "$INVENTORY" "$TMP" > "$TMP/probes" <<'JS'
++const fs = require('fs');
++const path = require('path');
++const [inventory, tmp] = process.argv.slice(2);
++let text;
++try {
++    text = fs.readFileSync(inventory, 'utf8');
++} catch {
++    process.stderr.write(`ready-health: cannot read the inventory file ${inventory}\n`);
++    process.exit(1);
++}
++let body;
++try {
++    body = JSON.parse(text);
++} catch {
++    process.stderr.write(`ready-health: the inventory ${inventory} is not JSON\n`);
++    process.exit(1);
++}
++const services = body && body.services;
++if (!services || typeof services !== 'object' || Array.isArray(services)) {
++    process.stderr.write(`ready-health: the inventory ${inventory} has no services object\n`);
++    process.exit(1);
++}
++const ID = /^[a-z][a-z0-9-]{0,39}$/;
++const HEADER = /^[A-Za-z0-9!#$%&'*+.^_`|~-]+$/;
++const quoted = (v) => '"' + String(v).replace(/\\/g, '\\\\').replace(/"/g, '\\"') + '"';
++let n = 0;
++for (const id of Object.keys(services).sort()) {
++    const svc = services[id];
++    if (!svc || typeof svc !== 'object' || !svc.ready || typeof svc.ready.url !== 'string' || !svc.ready.url) continue;
++    if (!ID.test(id)) {
++        process.stderr.write(`ready-health: skip "${id}": not a service id\n`);
++        continue;
++    }
++    let url;
++    try {
++        url = new URL(svc.ready.url);
++    } catch {
++        process.stderr.write(`ready-health: skip ${id}: ready.url is not a URL\n`);
++        continue;
++    }
++    if (!['http:', 'https:'].includes(url.protocol) || url.username || url.password) {
++        process.stderr.write(`ready-health: skip ${id}: ready.url must be an http(s) URL without credentials\n`);
++        continue;
++    }
++    const lines = [`url = ${quoted(svc.ready.url)}`];
++    const headers = svc.ready.headers && typeof svc.ready.headers === 'object' && !Array.isArray(svc.ready.headers) ? svc.ready.headers : {};
++    for (const [name, value] of Object.entries(headers)) {
++        if (!HEADER.test(name) || typeof value !== 'string' || /[\r\n\0]/.test(value)) {
++            process.stderr.write(`ready-health: skip ${id} header ${JSON.stringify(name)}: not a header name and a single-line value\n`);
++            continue;
++        }
++        lines.push(`header = ${quoted(`${name}: ${value}`)}`);
++    }
++    const file = path.join(tmp, `probe-${n++}.conf`);
++    fs.writeFileSync(file, `${lines.join('\n')}\n`, { mode: 0o600 });
++    process.stdout.write(`${id}\t${file}\n`);
++}
++if (n === 0) {
++    process.stderr.write(`ready-health: ${inventory} names no service with a ready URL\n`);
++    process.exit(1);
++}
++JS
++then
++    echo "ready-health: the inventory could not be read; keeping $OUT" >&2
++    exit 2
++fi
++
++LINES=()
++FAILED=0
++while IFS=$'\t' read -r id cfg; do
++    [ -n "$id" ] || continue
++    code=$(curl -sS -o /dev/null --max-time 5 --connect-timeout 5 -w '%{http_code}' -K "$cfg" 2>/dev/null) || code=000
++    case "$code" in
++        2??) value=1 ;;
++        *)   value=0; FAILED=$((FAILED + 1)) ;;
++    esac
++    LINES+=("openvibe_service_ready{service=\"$id\"} $value")
++done < "$TMP/probes"
++
++LINES+=("openvibe_service_ready_last_run_timestamp $(date +%s)")
++
++if [ -d "$DIR" ]; then
++    TMPFILE=$OUT.tmp.$$
++    if ! {
++        echo '# HELP openvibe_service_ready 1 when the service ready URL answered 2xx within 5 s, 0 when it did not.'
++        echo '# TYPE openvibe_service_ready gauge'
++        echo '# HELP openvibe_service_ready_last_run_timestamp Unix time of the last readiness check that ran to the end.'
++        echo '# TYPE openvibe_service_ready_last_run_timestamp gauge'
++        printf '%s\n' "${LINES[@]}"
++    } > "$TMPFILE"; then
++        rm -f "$TMPFILE"
++        echo "ready-health: could not write $TMPFILE" >&2
++        exit 2
++    fi
++    chmod 0644 "$TMPFILE" || { rm -f "$TMPFILE"; echo "ready-health: could not chmod $TMPFILE" >&2; exit 2; }
++    if ! mv -f "$TMPFILE" "$OUT"; then
++        rm -f "$TMPFILE"
++        echo "ready-health: could not rename $TMPFILE to $OUT" >&2
++        exit 2
++    fi
++fi
++
++printf '%s\n' "${LINES[@]}"
++
++[ "$FAILED" -eq 0 ] || exit 1
++exit 0
ddiff --git a/deploy/systemd/openvibe-readyhealth.service b/deploy/systemd/openvibe-readyhealth.service
nnew file mode 100644
iindex 0000000..a34e339
---- /dev/null
++++ b/deploy/systemd/openvibe-readyhealth.service
@@@ -0,0 +1,17 @@
++[Unit]
++Description=OpenVibe readiness probe: GET every service's ready URL (docs/ready-health.md)
++Documentation=https://github.com/OpenVibers/OpenVibe.Host/blob/main/docs/ready-health.md
++After=network-online.target
++Wants=network-online.target
++
++[Service]
++Type=oneshot
++# Root reads the root-owned inventory and writes node_exporter's textfile collector; nothing else is touched.
++User=root
++ExecStart=/usr/local/lib/openvibe-host/deploy/scripts/ready-health.sh
++Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
++Environment=OVHOST_INVENTORY=/etc/openvibe/host.json
++Nice=10
++TimeoutStartSec=10min
++PrivateTmp=true
++ProtectSystem=full
ddiff --git a/deploy/systemd/openvibe-readyhealth.timer b/deploy/systemd/openvibe-readyhealth.timer
nnew file mode 100644
iindex 0000000..ef922e5
---- /dev/null
++++ b/deploy/systemd/openvibe-readyhealth.timer
@@@ -0,0 +1,10 @@
++[Unit]
++Description=Check that every OpenVibe service's ready URL answers, every 5 minutes (docs/ready-health.md)
++
++[Timer]
++OnBootSec=5min
++OnUnitActiveSec=5min
++AccuracySec=30s
++
++[Install]
++WantedBy=timers.target
ddiff --git a/docs/operator-alerts.md b/docs/operator-alerts.md
iindex 104c24d..cfe28db 100644
---- a/docs/operator-alerts.md
++++ b/docs/operator-alerts.md
@@@ -3,8 +3,9 @@
  Roadmap WS-H task 11. The host's Prometheus evaluates the rules in
  [deploy/prometheus/openvibe-rules.yml](../deploy/prometheus/openvibe-rules.yml): targets down, error
  rates, disk and memory, backups, deploy drift, the developer path, the Tools job proof, the browser check
--of each release, the sign-in canary of every site ([auth-health.md](auth-health.md)) and the relay itself.
--Until 2026-09-26 nothing delivered them.
++of each release, the sign-in canary of every site ([auth-health.md](auth-health.md)), the readiness of
++every service ([ready-health.md](ready-health.md)) and the relay itself. Until 2026-09-26 nothing
++delivered them.
  
  ## The path
  
@@@ -79,7 +80,8 @@ systemctl list-timers openvibe-alerts.timer
  sudo cp /usr/local/lib/openvibe-host/deploy/systemd/openvibe-alerts.{service,timer} /etc/systemd/system/
  sudo cp /usr/local/lib/openvibe-host/deploy/systemd/openvibe-browsercheck.{service,timer} /etc/systemd/system/
  sudo cp /usr/local/lib/openvibe-host/deploy/systemd/openvibe-authhealth.{service,timer} /etc/systemd/system/
--sudo systemctl daemon-reload && sudo systemctl enable --now openvibe-alerts.timer openvibe-browsercheck.timer openvibe-authhealth.timer
++sudo cp /usr/local/lib/openvibe-host/deploy/systemd/openvibe-readyhealth.{service,timer} /etc/systemd/system/
++sudo systemctl daemon-reload && sudo systemctl enable --now openvibe-alerts.timer openvibe-browsercheck.timer openvibe-authhealth.timer openvibe-readyhealth.timer
  # Chrome and the check account (once):
  curl -fsSL https://dl.google.com/linux/linux_signing_key.pub | sudo gpg --dearmor -o /usr/share/keyrings/google-chrome.gpg
  echo "deb [arch=amd64 signed-by=/usr/share/keyrings/google-chrome.gpg] https://dl.google.com/linux/chrome/deb/ stable main" | sudo tee /etc/apt/sources.list.d/google-chrome.list
ddiff --git a/docs/ready-health.md b/docs/ready-health.md
nnew file mode 100644
iindex 0000000..656861f
---- /dev/null
++++ b/docs/ready-health.md
@@@ -0,0 +1,77 @@
++# Readiness check
++
++A readiness check for every OpenVibe service on the production host. On 2026-11-05 openvibe.games
++answered its `/api/ready` with 503 for eight hours while its process and `/metrics` stayed up: `TargetDown`
++never fired and nobody knew. This notices.
++
++`deploy/scripts/ready-health.sh` (openvibe-readyhealth.timer, every 5 minutes) reads the host inventory
++(`/etc/openvibe/host.json`), and for every service with a `ready.url` — the same field `ovhost deploy`
++polls — GETs that URL with the entry's `ready.headers` and a 5 s timeout. It writes node_exporter's
++textfile collector in one rename:
++
++| Answer | Means | Metric |
++|---|---|---|
++| 2xx | the service answers its readiness endpoint | `openvibe_service_ready{service="<id>"} 1` |
++| any other status (a 500, a 503, a redirect), or no answer at all | not ready | `openvibe_service_ready{service="<id>"} 0` |
++| a service with no `ready.url` | nothing declared to judge | no line |
++
++Every completed run also writes `openvibe_service_ready_last_run_timestamp` (Unix seconds), so a check
++that stops running is noticed too.
++
++## The run
++
++`deploy/scripts/ready-health.sh` reads the inventory named by `OVHOST_INVENTORY` (default
++`/etc/openvibe/host.json`), one service at a time, and writes
++`/var/lib/prometheus/node-exporter/openvibe_service_ready.prom` in one rename, so Prometheus never reads
++half a file. It prints the lines it wrote.
++
++- **The inventory cannot be read** (missing, not JSON, no `services`, or no service with a ready URL): it
++  exits 2 and leaves the previous file alone. An inventory that cannot be checked is not green.
++- **No collector directory**: it writes nothing (the lines are still printed), so it is harmless off the
++  host.
++- **Exit code**: 0 every service is ready · 1 one service is 0 · 2 the inventory could not be read.
++- **Secrets**: it reads no env file and prints nothing from one. A `ready.headers` value (for example the
++  Tools gateway's `Host`) reaches curl through a mode-0600 temp file and is never echoed; the metric is a
++  service id and 0 or 1.
++
++After a deploy, or to run it by hand now:
++
++```bash
++sudo systemctl start openvibe-readyhealth.service        # every service, now
++sudo /usr/local/lib/openvibe-host/deploy/scripts/ready-health.sh   # the lines, without the unit
++```
++
++## The alert
++
++Host keeps its rules in [deploy/prometheus/openvibe-rules.yml](../deploy/prometheus/openvibe-rules.yml);
++nothing installs them automatically, so the copy in [Install](#install-once) is how a rule — this one and
++the others — reaches `/etc/prometheus/rules/openvibe.yml`.
++
++- `OpenVibeServiceNotReady` fires on `openvibe_service_ready == 0` for 15 minutes, severity `page`:
++  > `games` has not been ready for 15 minutes (read its /api/ready; sudo ovhost status games)
++- `OpenVibeServiceReadyMissed` fires when `openvibe_service_ready_last_run_timestamp` is more than
++  20 minutes old (or is absent), severity `page`: the probe itself has stopped, so every service would
++  look fine.
++
++Both travel the usual path ([operator-alerts.md](operator-alerts.md)).
++
++## Install (once)
++
++```bash
++sudo cp /usr/local/lib/openvibe-host/deploy/systemd/openvibe-readyhealth.{service,timer} /etc/systemd/system/
++sudo systemctl daemon-reload && sudo systemctl enable --now openvibe-readyhealth.timer
++sudo cp /usr/local/lib/openvibe-host/deploy/prometheus/openvibe-rules.yml /etc/prometheus/rules/openvibe.yml
++sudo promtool check rules /etc/prometheus/rules/openvibe.yml && sudo systemctl reload prometheus
++```
++
++The script lives in the `ovhost` CLI's own checkout (`/usr/local/lib/openvibe-host`), which
++`sudo ovhost self-update` brings up to `main`; the unit runs it from there.
++
++## What it cannot see
++
++- A 2xx says the readiness endpoint answered, not that the service works: a readiness route that forgets
++  to check its database still looks green. It complements `/metrics` and `TargetDown`, it does not
++  replace them.
++- A service without `ready.url` is outside the check, and a ready URL that takes longer than 5 s counts
++  as not ready.
++- The inventory is root-owned and never committed; this check reads it and writes only the textfile.
ddiff --git a/test/ready-health.test.js b/test/ready-health.test.js
nnew file mode 100644
iindex 0000000..1321cfd
---- /dev/null
++++ b/test/ready-health.test.js
@@@ -0,0 +1,216 @@
++'use strict';
++/**
++ * The readiness check (deploy/scripts/ready-health.sh, docs/ready-health.md): every service in the host
++ * inventory with a ready.url gets one GET with its ready.headers — 1 for a 2xx, 0 for any other answer or
++ * no answer at all, no line without a ready URL — plus the last-run timestamp. The textfile collector's
++ * file is replaced in one rename, and an inventory it cannot read leaves the previous file alone, so a
++ * broken list is never green. The script runs for real here; the services are local HTTP servers.
++ */
++const assert = require('assert');
++const fs = require('fs');
++const http = require('http');
++const os = require('os');
++const path = require('path');
++const { execFile, spawnSync } = require('child_process');
++const { test, runTests } = require('./helpers');
++
++const SCRIPT = path.join(__dirname, '..', 'deploy', 'scripts', 'ready-health.sh');
++const PROM = 'openvibe_service_ready.prom';
++
++function service(status, { location } = {}) {
++    const s = http.createServer((req, res) => {
++        s.hits.push({ url: req.url, headers: req.headers });
++        res.statusCode = status;
++        if (location) res.setHeader('location', location);
++        res.end(status >= 500 ? 'not ready' : 'ok');
++    });
++    s.hits = [];
++    return new Promise((resolve) => s.listen(0, '127.0.0.1', () => resolve(s)));
++}
++
++const originOf = (s) => `http://127.0.0.1:${s.address().port}`;
++
++/** A port nothing listens on: the check is asked to meet a service that does not answer. */
++function closedPort() {
++    return new Promise((resolve) => {
++        const s = http.createServer();
++        s.listen(0, '127.0.0.1', () => { const port = s.address().port; s.close(() => resolve(port)); });
++    });
++}
++
++function base() {
++    return fs.mkdtempSync(path.join(os.tmpdir(), 'ready-health-'));
++}
++
++function textfileDir(dirPath) {
++    const d = path.join(dirPath, 'textfile');
++    fs.mkdirSync(d);
++    return d;
++}
++
++function inventoryFile(dirPath, services) {
++    const file = path.join(dirPath, 'host.json');
++    fs.writeFileSync(file, JSON.stringify({ host: 'test-host', services }));
++    return file;
++}
++
++function inventoryRaw(dirPath, body) {
++    const file = path.join(dirPath, 'host.json');
++    fs.writeFileSync(file, body);
++    return file;
++}
++
++// execFile, not spawnSync: the services are servers in this process, and a blocked event loop could not
++// answer them (every service would meet a timeout instead).
++function run(inventory, textfile) {
++    return new Promise((resolve) => {
++        execFile('bash', [SCRIPT], {
++            timeout: 120000,
++            env: { ...process.env, OVHOST_INVENTORY: inventory, TEXTFILE_DIR: textfile },
++        }, (err, stdout, stderr) => resolve({ status: err ? (err.code == null ? null : err.code) : 0, stdout, stderr }));
++    });
++}
++
++const prom = (textfile) => {
++    try { return fs.readFileSync(path.join(textfile, PROM), 'utf8'); } catch { return null; }
++};
++
++runTests([
++    test('every service with a ready URL is probed once; 2xx is 1, everything else 0, headers are sent', async () => {
++        const ok = await service(200);
++        const created = await service(201);
++        const refused = await service(503);
++        const broken = await service(500);
++        const redirect = await service(302, { location: '/sign-in' });
++        const gate = await service(401);
++        const tools = await service(200);
++        const dead = await closedPort();
++        const dir = base();
++        const textfile = textfileDir(dir);
++        const inventory = inventoryFile(dir, {
++            live: { ready: { url: `${originOf(ok)}/api/ready` } },
++            codes: { ready: { url: `${originOf(created)}/api/ready` } },
++            cam: { ready: { url: `${originOf(refused)}/api/ready` } },
++            chat: { ready: { url: `${originOf(broken)}/api/ready` } },
++            blog: { ready: { url: `${originOf(redirect)}/api/ready` } },
++            ai: { ready: { url: `${originOf(gate)}/api/ready` } },
++            work: { ready: { url: `http://127.0.0.1:${dead}/api/ready` } },
++            tools: {
++                ready: {
++                    url: `${originOf(tools)}/api/ready`,
++                    headers: { Host: 'openvibe.tools', Authorization: 'Bearer test-secret' },
++                },
++            },
++            sites: { repo: '/opt/openvibe.sites' },
++        });
++        const r = await run(inventory, textfile);
++        for (const s of [ok, created, refused, broken, redirect, gate, tools]) s.close();
++        assert.strictEqual(r.status, 1, r.stderr);
++        for (const s of [ok, created, refused, broken, redirect, gate, tools]) {
++            assert.deepStrictEqual(s.hits.map((h) => h.url), ['/api/ready'], 'exactly one probe per service');
++        }
++        assert.strictEqual(tools.hits[0].headers.host, 'openvibe.tools', 'the inventory ready.headers are sent');
++        assert.strictEqual(tools.hits[0].headers.authorization, 'Bearer test-secret');
++        const file = prom(textfile);
++        assert.match(file, /^# TYPE openvibe_service_ready gauge$/m);
++        for (const id of ['live', 'codes', 'tools']) assert.match(file, new RegExp(`^openvibe_service_ready\\{service="${id}"\\} 1$`, 'm'));
++        for (const id of ['cam', 'chat', 'blog', 'ai', 'work']) assert.match(file, new RegExp(`^openvibe_service_ready\\{service="${id}"\\} 0$`, 'm'), id);
++        assert.ok(!/service="sites"/.test(file), 'a service with no ready URL gets no line');
++        assert.match(file, /^openvibe_service_ready_last_run_timestamp \d+$/m);
++        const lines = r.stdout.trim().split('\n');
++        assert.deepStrictEqual(lines.slice(0, 8), [
++            'openvibe_service_ready{service="ai"} 0',
++            'openvibe_service_ready{service="blog"} 0',
++            'openvibe_service_ready{service="cam"} 0',
++            'openvibe_service_ready{service="chat"} 0',
++            'openvibe_service_ready{service="codes"} 1',
++            'openvibe_service_ready{service="live"} 1',
++            'openvibe_service_ready{service="tools"} 1',
++            'openvibe_service_ready{service="work"} 0',
++        ]);
++        assert.match(lines[8], /^openvibe_service_ready_last_run_timestamp \d+$/);
++        assert.ok(Math.abs(Date.now() / 1000 - Number(lines[8].split(' ')[1])) < 60, 'the run timestamp is now');
++        assert.ok(!/test-secret/.test(r.stdout) && !/test-secret/.test(r.stderr), 'a header secret is never printed');
++        assert.deepStrictEqual(fs.readdirSync(textfile), [PROM], 'no temp file is left behind');
++    }),
++    test('all 2xx exits 0, a service without a ready URL is skipped, and the file is replaced in one rename', async () => {
++        const live = await service(200);
++        const blog = await service(204);
++        const dir = base();
++        const textfile = textfileDir(dir);
++        const inventory = inventoryFile(dir, {
++            live: { ready: { url: originOf(live) } },
++            blog: { ready: { url: `${originOf(blog)}/api/ready` } },
++            sites: { repo: '/opt/openvibe.sites' },
++        });
++        let r = await run(inventory, textfile);
++        assert.strictEqual(r.status, 0, r.stderr);
++        assert.match(prom(textfile), /^openvibe_service_ready\{service="live"\} 1$/m);
++        assert.match(prom(textfile), /^openvibe_service_ready\{service="blog"\} 1$/m);
++        assert.ok(!/service="sites"/.test(prom(textfile)));
++        const first = fs.statSync(path.join(textfile, PROM)).ino;
++        const firstTs = Number(prom(textfile).match(/last_run_timestamp (\d+)/)[1]);
++        await new Promise((resolve) => setTimeout(resolve, 1100));
++        r = await run(inventory, textfile);
++        assert.strictEqual(r.status, 0, r.stderr);
++        const second = fs.statSync(path.join(textfile, PROM)).ino;
++        const secondTs = Number(prom(textfile).match(/last_run_timestamp (\d+)/)[1]);
++        assert.notStrictEqual(second, first, 'a rename, not a rewrite in place');
++        assert.ok(secondTs > firstTs, 'the run timestamp moves');
++        assert.deepStrictEqual([live.hits.length, blog.hits.length], [2, 2]);
++        live.close();
++        blog.close();
++    }),
++    test('an inventory it cannot read, is not JSON, or names no ready URL keeps the previous file', async () => {
++        const dir = base();
++        const textfile = textfileDir(dir);
++        fs.writeFileSync(path.join(textfile, PROM), 'openvibe_service_ready{service="live"} 1\n');
++        const before = prom(textfile);
++        let r = await run(path.join(dir, 'absent.json'), textfile);
++        assert.strictEqual(r.status, 2);
++        assert.match(r.stderr, /cannot read the inventory file/);
++        r = await run(inventoryRaw(dir, '<html>502</html>'), textfile);
++        assert.strictEqual(r.status, 2);
++        assert.match(r.stderr, /is not JSON/);
++        r = await run(inventoryFile(dir, { live: { repo: '/opt/openvibe.live' } }), textfile);
++        assert.strictEqual(r.status, 2);
++        assert.match(r.stderr, /names no service with a ready URL/);
++        r = await run(inventoryFile(dir, {}), textfile);
++        assert.strictEqual(r.status, 2);
++        assert.match(r.stderr, /names no service with a ready URL/);
++        assert.strictEqual(prom(textfile), before, 'an inventory that cannot be checked is not green');
++    }),
++    test('without the collector directory nothing is written, and the lines are still printed', async () => {
++        const ok = await service(200);
++        const dir = base();
++        const inventory = inventoryFile(dir, { live: { ready: { url: originOf(ok) } } });
++        const absent = path.join(dir, 'no-collector-here');
++        const r = await run(inventory, absent);
++        assert.strictEqual(r.status, 0, r.stderr);
++        assert.match(r.stdout, /^openvibe_service_ready\{service="live"\} 1$/m);
++        assert.match(r.stdout, /^openvibe_service_ready_last_run_timestamp \d+$/m);
++        assert.ok(!fs.existsSync(path.join(absent, PROM)), 'nothing to write to, nothing written');
++        ok.close();
++    }),
++    test('the timer is every 5 minutes, the rules page on a service that stays not ready or a check that stops', () => {
++        const root = path.join(__dirname, '..');
++        const timer = fs.readFileSync(path.join(root, 'deploy', 'systemd', 'openvibe-readyhealth.timer'), 'utf8');
++        assert.match(timer, /^OnBootSec=5min$/m);
++        assert.match(timer, /^OnUnitActiveSec=5min$/m);
++        assert.match(timer, /^WantedBy=timers\.target$/m);
++        const unit = fs.readFileSync(path.join(root, 'deploy', 'systemd', 'openvibe-readyhealth.service'), 'utf8');
++        assert.match(unit, /^ExecStart=\/usr\/local\/lib\/openvibe-host\/deploy\/scripts\/ready-health\.sh$/m);
++        assert.match(unit, /^Environment=OVHOST_INVENTORY=\/etc\/openvibe\/host\.json$/m);
++        assert.match(unit, /^User=root$/m);
++        const rules = fs.readFileSync(path.join(root, 'deploy', 'prometheus', 'openvibe-rules.yml'), 'utf8');
++        assert.match(rules, /- alert: OpenVibeServiceNotReady\n\s+expr: openvibe_service_ready == 0\n\s+for: 15m\n\s+labels: \{ severity: page \}/);
++        const notReady = rules.slice(rules.indexOf('alert: OpenVibeServiceNotReady'), rules.indexOf('alert: OpenVibeServiceReadyMissed'));
++        assert.match(notReady, /summary: ".*\$labels\.service.*\/api\/ready/);
++        assert.match(rules, /- alert: OpenVibeServiceReadyMissed\n\s+expr: time\(\) - openvibe_service_ready_last_run_timestamp > 20 \* 60 or absent\(openvibe_service_ready_last_run_timestamp\)\n\s+for: 5m\n\s+labels: \{ severity: page \}/);
++        const doc = fs.readFileSync(path.join(root, 'docs', 'ready-health.md'), 'utf8');
++        assert.match(doc, /openvibe_service_ready_last_run_timestamp/);
++        assert.match(doc, /openvibe-readyhealth\.timer/);
++        const r = spawnSync('bash', ['-n', SCRIPT], { encoding: 'utf8' });
++        assert.strictEqual(r.status, 0, r.stderr);
++    }),
++]);