openvibers/openvibe.host Commits

openvibe-toolsjob.timer (WS-L task 4): the Tools job proof every six hours as principal probe; devpath-metrics.sh takes METRIC_PREFIX; alerts OpenVibeToolsJobProofFailed/Missed

OP OpenVibers committed 25 Sept 2026 · 0cadec3e71 · 5 files, +101 −14


Changes

ddiff --git a/deploy/prometheus/openvibe-rules.yml b/deploy/prometheus/openvibe-rules.yml
iindex 11e0b9a..04fad1a 100644
---- a/deploy/prometheus/openvibe-rules.yml
++++ b/deploy/prometheus/openvibe-rules.yml
@@@ -64,3 +64,16 @@ groups:
labels: { severity: ticket }
annotations:
summary: "No passing developer-path run for more than 30 hours"
++ # The Tools job proof against production, every six hours (openvibe-toolsjob.timer; WS-L task 4).
++ - alert: OpenVibeToolsJobProofFailed
++ expr: openvibe_toolsjob_last_run_ok == 0
++ for: 5m
++ labels: { severity: ticket }
++ annotations:
++ summary: "The Tools job proof failed ({{ with query \"openvibe_toolsjob_steps_failed\" }}{{ . | first | value }}{{ end }} step(s)): a converter job, its Media result or its tools.job.* events did not come through"
++ - alert: OpenVibeToolsJobProofMissed
++ expr: time() - openvibe_toolsjob_last_success_timestamp_seconds > 13 * 3600
++ for: 30m
++ labels: { severity: ticket }
++ annotations:
++ summary: "No passing Tools job proof for more than 13 hours (two runs)"
ddiff --git a/deploy/systemd/openvibe-toolsjob.service b/deploy/systemd/openvibe-toolsjob.service
nnew file mode 100644
iindex 0000000..66dc079
---- /dev/null
++++ b/deploy/systemd/openvibe-toolsjob.service
@@@ -0,0 +1,26 @@
++[Unit]
++Description=OpenVibe Tools job proof against production (roadmap WS-L task 4): an SDK-only service client runs a converter job, reattaches its progress stream, reads the result from Media and finds its tools.job.* events
++Documentation=https://github.com/OpenVibers/OpenVibe.Host/blob/main/docs/developer-path.md#tools-job-proof
++After=network-online.target
++Wants=network-online.target
++
++[Service]
++Type=oneshot
++# The probe principal's client credentials (OV_OAUTH_CLIENT_ID/SECRET, written by Network's service-principal
++# setup) come from a root-only env file that systemd reads before dropping to the unprivileged user; the
++# script never prints them, and --result holds step names only.
++User=ubuntu
++EnvironmentFile=/etc/openvibe/probe.env
++Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
++# For the metrics script (ExecStopPost): which result file, which metric names.
++Environment=DEVPATH_RESULT=/var/lib/openvibe-devpath/tools-job.json METRIC_PREFIX=openvibe_toolsjob
++WorkingDirectory=/usr/local/lib/openvibe-examples
++StateDirectory=openvibe-devpath
++StateDirectoryMode=0755
++ExecStart=/usr/bin/env node scripts/tools-job-proof.js --result /var/lib/openvibe-devpath/tools-job.json
++# As root: the outcome for Prometheus (node_exporter's textfile collector), whatever the exit code.
++ExecStopPost=+/usr/local/lib/openvibe-host/scripts/devpath-metrics.sh
++TimeoutStartSec=15min
++NoNewPrivileges=true
++PrivateTmp=true
++ProtectSystem=full
ddiff --git a/deploy/systemd/openvibe-toolsjob.timer b/deploy/systemd/openvibe-toolsjob.timer
nnew file mode 100644
iindex 0000000..5f3dc7d
---- /dev/null
++++ b/deploy/systemd/openvibe-toolsjob.timer
@@@ -0,0 +1,10 @@
++[Unit]
++Description=Run the OpenVibe Tools job proof against production every six hours (WS-L task 4)
++
++[Timer]
++OnCalendar=*-*-* 00/6:25:00 UTC
++RandomizedDelaySec=10min
++Persistent=true
++
++[Install]
++WantedBy=timers.target
ddiff --git a/docs/developer-path.md b/docs/developer-path.md
iindex 1d203a8..8ec62a4 100644
---- a/docs/developer-path.md
++++ b/docs/developer-path.md
@@@ -26,3 +26,37 @@ systemctl enable --now openvibe-devpath.timer
```
Run now: `systemctl start openvibe-devpath.service`; read: `journalctl -u openvibe-devpath` (masked).
++
++## Tools job proof
++
++`openvibe-toolsjob.timer` runs OpenVibe.Examples' `scripts/tools-job-proof.js` every six hours (xx:25 UTC,
++up to 10 minutes later; roadmap WS-L task 4). It is the end-to-end proof of Tools jobs, using the SDK only. The `probe` service
++principal gets tokens for `tools.job.create`, `tools.job.read` and `events.event.read`. It then:
++
++1. submits an `img.process` job that converts a PNG to WebP;
++2. drops the progress stream after its first event and reattaches with `Last-Event-ID`, as the UI does;
++3. downloads the result, which is stored in Media;
++4. finds the job's `tools.job.created`, `started` and `succeeded` events in the Events store.
++
++A service principal runs it, not a developer app, because sandbox jobs are never announced to Events or
++copied to Media.
++
++| Piece | Where |
++|---|---|
++| Checkout | the same `/usr/local/lib/openvibe-examples` |
++| Identity | `/etc/openvibe/probe.env` (root, 0600): `OV_OAUTH_CLIENT_ID`, `OV_OAUTH_CLIENT_SECRET` of principal `probe`; grants in OpenVibe.Network `server/identity/principals.js` |
++| Units | `deploy/systemd/openvibe-toolsjob.service` + `.timer` (runs as `ubuntu`) |
++| Result | `/var/lib/openvibe-devpath/tools-job.json`: step names and ok only (shown on openvibe.network/status) |
++| Metrics | `scripts/devpath-metrics.sh` with `METRIC_PREFIX=openvibe_toolsjob` → `openvibe_toolsjob.prom` |
++| Alerts | `OpenVibeToolsJobProofFailed`, `OpenVibeToolsJobProofMissed` (13 h, two runs) |
++
++Install (once, as root, in `/opt/openvibe.network`, then here):
++
++```sh
++node server/setup/service-principal.js create probe --env-file /etc/openvibe/probe.env # secret never printed
++systemctl restart openvibe-network # seeds probe's default grants
++install -m 644 deploy/systemd/openvibe-toolsjob.{service,timer} /etc/systemd/system/ && systemctl daemon-reload
++systemctl enable --now openvibe-toolsjob.timer
++```
++
++Run now: `systemctl start openvibe-toolsjob.service`; read: `journalctl -u openvibe-toolsjob`.
ddiff --git a/scripts/devpath-metrics.sh b/scripts/devpath-metrics.sh
iindex 61a50ee..b03b9ed 100755
---- a/scripts/devpath-metrics.sh
++++ b/scripts/devpath-metrics.sh
@@@ -1,34 +1,38 @@
#!/usr/bin/env bash
--# The developer path's outcome for Prometheus (openvibe-devpath.service ExecStopPost, as root; WS-N task 1):
--# node_exporter's textfile collector reads /var/lib/prometheus/node-exporter/openvibe_devpath.prom.
--# openvibe_devpath_last_run_timestamp_seconds, openvibe_devpath_last_run_ok,
--# openvibe_devpath_last_success_timestamp_seconds (kept from the previous file when this run failed),
--# openvibe_devpath_steps_failed
++# A scheduled proof's outcome for Prometheus (ExecStopPost, as root): the developer path
++# (openvibe-devpath.service, WS-N task 1) and, with METRIC_PREFIX=openvibe_toolsjob and its own
++# DEVPATH_RESULT, the Tools job proof (openvibe-toolsjob.service, WS-L task 4).
++# node_exporter's textfile collector reads /var/lib/prometheus/node-exporter/<prefix>.prom.
++# <prefix>_last_run_timestamp_seconds, <prefix>_last_run_ok,
++# <prefix>_last_success_timestamp_seconds (kept from the previous file when this run failed),
++# <prefix>_steps_failed
# A run that refused to start or wrote no result (exit != 0, no fresh last.json) counts as failed.
set -euo pipefail
RESULT=${DEVPATH_RESULT:-/var/lib/openvibe-devpath/last.json}
++PREFIX=${METRIC_PREFIX:-openvibe_devpath}
++[[ "$PREFIX" =~ ^[a-z_]+$ ]] || { echo "bad METRIC_PREFIX" >&2; exit 1; }
DIR=${TEXTFILE_DIR:-/var/lib/prometheus/node-exporter}
[ -d "$DIR" ] || exit 0
--OUT="$DIR/openvibe_devpath.prom"
--node - "$RESULT" "$OUT" "${EXIT_STATUS:-}" <<'JS'
++OUT="$DIR/$PREFIX.prom"
++node - "$RESULT" "$OUT" "${EXIT_STATUS:-}" "$PREFIX" <<'JS'
const fs = require('fs');
--const [file, out, exitStatus] = process.argv.slice(2);
++const [file, out, exitStatus, p] = process.argv.slice(2);
const now = Math.floor(Date.now() / 1000);
let rec = null;
try { rec = JSON.parse(fs.readFileSync(file, 'utf8')); } catch { rec = null; }
const fresh = rec && Date.parse(rec.finished_at) / 1000 > now - 3600;
const ok = !!(fresh && rec.ok && (exitStatus === '' || exitStatus === '0'));
let lastSuccess = null;
--try { const m = /^openvibe_devpath_last_success_timestamp_seconds (\d+)$/m.exec(fs.readFileSync(out, 'utf8')); if (m) lastSuccess = Number(m[1]); } catch { /* first run */ }
++try { const m = new RegExp(`^${p}_last_success_timestamp_seconds (\\d+)$`, 'm').exec(fs.readFileSync(out, 'utf8')); if (m) lastSuccess = Number(m[1]); } catch { /* first run */ }
if (ok) lastSuccess = Math.floor(Date.parse(rec.finished_at) / 1000);
const failed = fresh ? rec.steps.filter((s) => !s.ok && !s.skipped).length : 1;
const lines = [
-- '# HELP openvibe_devpath_last_run_timestamp_seconds When the last developer-path run ended.',
-- '# TYPE openvibe_devpath_last_run_timestamp_seconds gauge', `openvibe_devpath_last_run_timestamp_seconds ${fresh ? Math.floor(Date.parse(rec.finished_at) / 1000) : now}`,
-- '# HELP openvibe_devpath_last_run_ok 1 when every step passed.', '# TYPE openvibe_devpath_last_run_ok gauge', `openvibe_devpath_last_run_ok ${ok ? 1 : 0}`,
-- '# HELP openvibe_devpath_steps_failed Steps that failed in the last run.', '# TYPE openvibe_devpath_steps_failed gauge', `openvibe_devpath_steps_failed ${failed}`,
++ `# HELP ${p}_last_run_timestamp_seconds When the last run ended.`,
++ `# TYPE ${p}_last_run_timestamp_seconds gauge`, `${p}_last_run_timestamp_seconds ${fresh ? Math.floor(Date.parse(rec.finished_at) / 1000) : now}`,
++ `# HELP ${p}_last_run_ok 1 when every step passed.`, `# TYPE ${p}_last_run_ok gauge`, `${p}_last_run_ok ${ok ? 1 : 0}`,
++ `# HELP ${p}_steps_failed Steps that failed in the last run.`, `# TYPE ${p}_steps_failed gauge`, `${p}_steps_failed ${failed}`,
];
--if (lastSuccess) lines.push('# HELP openvibe_devpath_last_success_timestamp_seconds When the last fully passing run ended.', '# TYPE openvibe_devpath_last_success_timestamp_seconds gauge', `openvibe_devpath_last_success_timestamp_seconds ${lastSuccess}`);
++if (lastSuccess) lines.push(`# HELP ${p}_last_success_timestamp_seconds When the last fully passing run ended.`, `# TYPE ${p}_last_success_timestamp_seconds gauge`, `${p}_last_success_timestamp_seconds ${lastSuccess}`);
fs.writeFileSync(`${out}.tmp`, lines.join('\n') + '\n', { mode: 0o644 });
fs.renameSync(`${out}.tmp`, out);
JS