openvibers/openvibe.host Commits
openvibe-toolsjob.timer (WS-L task 4): the Tools job proof every six hours as principal probe; devpath-metrics.sh takes METRIC_PREFIX; alerts OpenVibeToolsJobProofFailed/Missed
Changes
@@@ -64,3 +64,16 @@ groups:labels: { severity: ticket }annotations:summary: "No passing developer-path run for more than 30 hours"++ # The Tools job proof against production, every six hours (openvibe-toolsjob.timer; WS-L task 4).++ - alert: OpenVibeToolsJobProofFailed++ expr: openvibe_toolsjob_last_run_ok == 0++ for: 5m++ labels: { severity: ticket }++ annotations:++ summary: "The Tools job proof failed ({{ with query \"openvibe_toolsjob_steps_failed\" }}{{ . | first | value }}{{ end }} step(s)): a converter job, its Media result or its tools.job.* events did not come through"++ - alert: OpenVibeToolsJobProofMissed++ expr: time() - openvibe_toolsjob_last_success_timestamp_seconds > 13 * 3600++ for: 30m++ labels: { severity: ticket }++ annotations:++ summary: "No passing Tools job proof for more than 13 hours (two runs)"nnew file mode 100644@@@ -0,0 +1,26 @@++[Unit]++Description=OpenVibe Tools job proof against production (roadmap WS-L task 4): an SDK-only service client runs a converter job, reattaches its progress stream, reads the result from Media and finds its tools.job.* events++Documentation=https://github.com/OpenVibers/OpenVibe.Host/blob/main/docs/developer-path.md#tools-job-proof++After=network-online.target++Wants=network-online.target++++[Service]++Type=oneshot++# The probe principal's client credentials (OV_OAUTH_CLIENT_ID/SECRET, written by Network's service-principal++# setup) come from a root-only env file that systemd reads before dropping to the unprivileged user; the++# script never prints them, and --result holds step names only.++User=ubuntu++EnvironmentFile=/etc/openvibe/probe.env++Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin++# For the metrics script (ExecStopPost): which result file, which metric names.++Environment=DEVPATH_RESULT=/var/lib/openvibe-devpath/tools-job.json METRIC_PREFIX=openvibe_toolsjob++WorkingDirectory=/usr/local/lib/openvibe-examples++StateDirectory=openvibe-devpath++StateDirectoryMode=0755++ExecStart=/usr/bin/env node scripts/tools-job-proof.js --result /var/lib/openvibe-devpath/tools-job.json++# As root: the outcome for Prometheus (node_exporter's textfile collector), whatever the exit code.++ExecStopPost=+/usr/local/lib/openvibe-host/scripts/devpath-metrics.sh++TimeoutStartSec=15min++NoNewPrivileges=true++PrivateTmp=true++ProtectSystem=fullnnew file mode 100644@@@ -0,0 +1,10 @@++[Unit]++Description=Run the OpenVibe Tools job proof against production every six hours (WS-L task 4)++++[Timer]++OnCalendar=*-*-* 00/6:25:00 UTC++RandomizedDelaySec=10min++Persistent=true++++[Install]++WantedBy=timers.target@@@ -26,3 +26,37 @@ systemctl enable --now openvibe-devpath.timer```Run now: `systemctl start openvibe-devpath.service`; read: `journalctl -u openvibe-devpath` (masked).++++## Tools job proof++++`openvibe-toolsjob.timer` runs OpenVibe.Examples' `scripts/tools-job-proof.js` every six hours (xx:25 UTC,++up to 10 minutes later; roadmap WS-L task 4). It is the end-to-end proof of Tools jobs, using the SDK only. The `probe` service++principal gets tokens for `tools.job.create`, `tools.job.read` and `events.event.read`. It then:++++1. submits an `img.process` job that converts a PNG to WebP;++2. drops the progress stream after its first event and reattaches with `Last-Event-ID`, as the UI does;++3. downloads the result, which is stored in Media;++4. finds the job's `tools.job.created`, `started` and `succeeded` events in the Events store.++++A service principal runs it, not a developer app, because sandbox jobs are never announced to Events or++copied to Media.++++| Piece | Where |++|---|---|++| Checkout | the same `/usr/local/lib/openvibe-examples` |++| Identity | `/etc/openvibe/probe.env` (root, 0600): `OV_OAUTH_CLIENT_ID`, `OV_OAUTH_CLIENT_SECRET` of principal `probe`; grants in OpenVibe.Network `server/identity/principals.js` |++| Units | `deploy/systemd/openvibe-toolsjob.service` + `.timer` (runs as `ubuntu`) |++| Result | `/var/lib/openvibe-devpath/tools-job.json`: step names and ok only (shown on openvibe.network/status) |++| Metrics | `scripts/devpath-metrics.sh` with `METRIC_PREFIX=openvibe_toolsjob` → `openvibe_toolsjob.prom` |++| Alerts | `OpenVibeToolsJobProofFailed`, `OpenVibeToolsJobProofMissed` (13 h, two runs) |++++Install (once, as root, in `/opt/openvibe.network`, then here):++++```sh++node server/setup/service-principal.js create probe --env-file /etc/openvibe/probe.env # secret never printed++systemctl restart openvibe-network # seeds probe's default grants++install -m 644 deploy/systemd/openvibe-toolsjob.{service,timer} /etc/systemd/system/ && systemctl daemon-reload++systemctl enable --now openvibe-toolsjob.timer++```++++Run now: `systemctl start openvibe-toolsjob.service`; read: `journalctl -u openvibe-toolsjob`.@@@ -1,34 +1,38 @@#!/usr/bin/env bash--# The developer path's outcome for Prometheus (openvibe-devpath.service ExecStopPost, as root; WS-N task 1):--# node_exporter's textfile collector reads /var/lib/prometheus/node-exporter/openvibe_devpath.prom.--# openvibe_devpath_last_run_timestamp_seconds, openvibe_devpath_last_run_ok,--# openvibe_devpath_last_success_timestamp_seconds (kept from the previous file when this run failed),--# openvibe_devpath_steps_failed++# A scheduled proof's outcome for Prometheus (ExecStopPost, as root): the developer path++# (openvibe-devpath.service, WS-N task 1) and, with METRIC_PREFIX=openvibe_toolsjob and its own++# DEVPATH_RESULT, the Tools job proof (openvibe-toolsjob.service, WS-L task 4).++# node_exporter's textfile collector reads /var/lib/prometheus/node-exporter/<prefix>.prom.++# <prefix>_last_run_timestamp_seconds, <prefix>_last_run_ok,++# <prefix>_last_success_timestamp_seconds (kept from the previous file when this run failed),++# <prefix>_steps_failed# A run that refused to start or wrote no result (exit != 0, no fresh last.json) counts as failed.set -euo pipefailRESULT=${DEVPATH_RESULT:-/var/lib/openvibe-devpath/last.json}++PREFIX=${METRIC_PREFIX:-openvibe_devpath}++[[ "$PREFIX" =~ ^[a-z_]+$ ]] || { echo "bad METRIC_PREFIX" >&2; exit 1; }DIR=${TEXTFILE_DIR:-/var/lib/prometheus/node-exporter}[ -d "$DIR" ] || exit 0--OUT="$DIR/openvibe_devpath.prom"--node - "$RESULT" "$OUT" "${EXIT_STATUS:-}" <<'JS'++OUT="$DIR/$PREFIX.prom"++node - "$RESULT" "$OUT" "${EXIT_STATUS:-}" "$PREFIX" <<'JS'const fs = require('fs');--const [file, out, exitStatus] = process.argv.slice(2);++const [file, out, exitStatus, p] = process.argv.slice(2);const now = Math.floor(Date.now() / 1000);let rec = null;try { rec = JSON.parse(fs.readFileSync(file, 'utf8')); } catch { rec = null; }const fresh = rec && Date.parse(rec.finished_at) / 1000 > now - 3600;const ok = !!(fresh && rec.ok && (exitStatus === '' || exitStatus === '0'));let lastSuccess = null;--try { const m = /^openvibe_devpath_last_success_timestamp_seconds (\d+)$/m.exec(fs.readFileSync(out, 'utf8')); if (m) lastSuccess = Number(m[1]); } catch { /* first run */ }++try { const m = new RegExp(`^${p}_last_success_timestamp_seconds (\\d+)$`, 'm').exec(fs.readFileSync(out, 'utf8')); if (m) lastSuccess = Number(m[1]); } catch { /* first run */ }if (ok) lastSuccess = Math.floor(Date.parse(rec.finished_at) / 1000);const failed = fresh ? rec.steps.filter((s) => !s.ok && !s.skipped).length : 1;const lines = [-- '# HELP openvibe_devpath_last_run_timestamp_seconds When the last developer-path run ended.',-- '# TYPE openvibe_devpath_last_run_timestamp_seconds gauge', `openvibe_devpath_last_run_timestamp_seconds ${fresh ? Math.floor(Date.parse(rec.finished_at) / 1000) : now}`,-- '# HELP openvibe_devpath_last_run_ok 1 when every step passed.', '# TYPE openvibe_devpath_last_run_ok gauge', `openvibe_devpath_last_run_ok ${ok ? 1 : 0}`,-- '# HELP openvibe_devpath_steps_failed Steps that failed in the last run.', '# TYPE openvibe_devpath_steps_failed gauge', `openvibe_devpath_steps_failed ${failed}`,++ `# HELP ${p}_last_run_timestamp_seconds When the last run ended.`,++ `# TYPE ${p}_last_run_timestamp_seconds gauge`, `${p}_last_run_timestamp_seconds ${fresh ? Math.floor(Date.parse(rec.finished_at) / 1000) : now}`,++ `# HELP ${p}_last_run_ok 1 when every step passed.`, `# TYPE ${p}_last_run_ok gauge`, `${p}_last_run_ok ${ok ? 1 : 0}`,++ `# HELP ${p}_steps_failed Steps that failed in the last run.`, `# TYPE ${p}_steps_failed gauge`, `${p}_steps_failed ${failed}`,];--if (lastSuccess) lines.push('# HELP openvibe_devpath_last_success_timestamp_seconds When the last fully passing run ended.', '# TYPE openvibe_devpath_last_success_timestamp_seconds gauge', `openvibe_devpath_last_success_timestamp_seconds ${lastSuccess}`);++if (lastSuccess) lines.push(`# HELP ${p}_last_success_timestamp_seconds When the last fully passing run ended.`, `# TYPE ${p}_last_success_timestamp_seconds gauge`, `${p}_last_success_timestamp_seconds ${lastSuccess}`);fs.writeFileSync(`${out}.tmp`, lines.join('\n') + '\n', { mode: 0o644 });fs.renameSync(`${out}.tmp`, out);JS
@@ -64,3 +64,16 @@ groups:
@@ -64,3 +64,16 @@ groups:
labels: { severity: ticket }
labels: { severity: ticket }
annotations:
annotations:
summary: "No passing developer-path run for more than 30 hours"
summary: "No passing developer-path run for more than 30 hours"
+ # The Tools job proof against production, every six hours (openvibe-toolsjob.timer; WS-L task 4).
+ - alert: OpenVibeToolsJobProofFailed
+ expr: openvibe_toolsjob_last_run_ok == 0
+ for: 5m
+ labels: { severity: ticket }
+ annotations:
+ summary: "The Tools job proof failed ({{ with query \"openvibe_toolsjob_steps_failed\" }}{{ . | first | value }}{{ end }} step(s)): a converter job, its Media result or its tools.job.* events did not come through"
+ - alert: OpenVibeToolsJobProofMissed
+ expr: time() - openvibe_toolsjob_last_success_timestamp_seconds > 13 * 3600
+ for: 30m
+ labels: { severity: ticket }
+ annotations:
+ summary: "No passing Tools job proof for more than 13 hours (two runs)"
new file mode 100644
new file mode 100644
@@ -0,0 +1,26 @@
@@ -0,0 +1,26 @@
+[Unit]
+Description=OpenVibe Tools job proof against production (roadmap WS-L task 4): an SDK-only service client runs a converter job, reattaches its progress stream, reads the result from Media and finds its tools.job.* events
+Documentation=https://github.com/OpenVibers/OpenVibe.Host/blob/main/docs/developer-path.md#tools-job-proof
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+# The probe principal's client credentials (OV_OAUTH_CLIENT_ID/SECRET, written by Network's service-principal
+# setup) come from a root-only env file that systemd reads before dropping to the unprivileged user; the
+# script never prints them, and --result holds step names only.
+User=ubuntu
+EnvironmentFile=/etc/openvibe/probe.env
+Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+# For the metrics script (ExecStopPost): which result file, which metric names.
+Environment=DEVPATH_RESULT=/var/lib/openvibe-devpath/tools-job.json METRIC_PREFIX=openvibe_toolsjob
+WorkingDirectory=/usr/local/lib/openvibe-examples
+StateDirectory=openvibe-devpath
+StateDirectoryMode=0755
+ExecStart=/usr/bin/env node scripts/tools-job-proof.js --result /var/lib/openvibe-devpath/tools-job.json
+# As root: the outcome for Prometheus (node_exporter's textfile collector), whatever the exit code.
+ExecStopPost=+/usr/local/lib/openvibe-host/scripts/devpath-metrics.sh
+TimeoutStartSec=15min
+NoNewPrivileges=true
+PrivateTmp=true
+ProtectSystem=full
new file mode 100644
new file mode 100644
@@ -0,0 +1,10 @@
@@ -0,0 +1,10 @@
+[Unit]
+Description=Run the OpenVibe Tools job proof against production every six hours (WS-L task 4)
+
+[Timer]
+OnCalendar=*-*-* 00/6:25:00 UTC
+RandomizedDelaySec=10min
+Persistent=true
+
+[Install]
+WantedBy=timers.target
@@ -26,3 +26,37 @@ systemctl enable --now openvibe-devpath.timer
@@ -26,3 +26,37 @@ systemctl enable --now openvibe-devpath.timer
```
```
Run now: `systemctl start openvibe-devpath.service`; read: `journalctl -u openvibe-devpath` (masked).
Run now: `systemctl start openvibe-devpath.service`; read: `journalctl -u openvibe-devpath` (masked).
+
+## Tools job proof
+
+`openvibe-toolsjob.timer` runs OpenVibe.Examples' `scripts/tools-job-proof.js` every six hours (xx:25 UTC,
+up to 10 minutes later; roadmap WS-L task 4). It is the end-to-end proof of Tools jobs, using the SDK only. The `probe` service
+principal gets tokens for `tools.job.create`, `tools.job.read` and `events.event.read`. It then:
+
+1. submits an `img.process` job that converts a PNG to WebP;
+2. drops the progress stream after its first event and reattaches with `Last-Event-ID`, as the UI does;
+3. downloads the result, which is stored in Media;
+4. finds the job's `tools.job.created`, `started` and `succeeded` events in the Events store.
+
+A service principal runs it, not a developer app, because sandbox jobs are never announced to Events or
+copied to Media.
+
+| Piece | Where |
+|---|---|
+| Checkout | the same `/usr/local/lib/openvibe-examples` |
+| Identity | `/etc/openvibe/probe.env` (root, 0600): `OV_OAUTH_CLIENT_ID`, `OV_OAUTH_CLIENT_SECRET` of principal `probe`; grants in OpenVibe.Network `server/identity/principals.js` |
+| Units | `deploy/systemd/openvibe-toolsjob.service` + `.timer` (runs as `ubuntu`) |
+| Result | `/var/lib/openvibe-devpath/tools-job.json`: step names and ok only (shown on openvibe.network/status) |
+| Metrics | `scripts/devpath-metrics.sh` with `METRIC_PREFIX=openvibe_toolsjob` → `openvibe_toolsjob.prom` |
+| Alerts | `OpenVibeToolsJobProofFailed`, `OpenVibeToolsJobProofMissed` (13 h, two runs) |
+
+Install (once, as root, in `/opt/openvibe.network`, then here):
+
+```sh
+node server/setup/service-principal.js create probe --env-file /etc/openvibe/probe.env # secret never printed
+systemctl restart openvibe-network # seeds probe's default grants
+install -m 644 deploy/systemd/openvibe-toolsjob.{service,timer} /etc/systemd/system/ && systemctl daemon-reload
+systemctl enable --now openvibe-toolsjob.timer
+```
+
+Run now: `systemctl start openvibe-toolsjob.service`; read: `journalctl -u openvibe-toolsjob`.
@@ -1,34 +1,38 @@
@@ -1,34 +1,38 @@
#!/usr/bin/env bash
#!/usr/bin/env bash
-# The developer path's outcome for Prometheus (openvibe-devpath.service ExecStopPost, as root; WS-N task 1):
-# node_exporter's textfile collector reads /var/lib/prometheus/node-exporter/openvibe_devpath.prom.
-# openvibe_devpath_last_run_timestamp_seconds, openvibe_devpath_last_run_ok,
-# openvibe_devpath_last_success_timestamp_seconds (kept from the previous file when this run failed),
-# openvibe_devpath_steps_failed
+# A scheduled proof's outcome for Prometheus (ExecStopPost, as root): the developer path
+# (openvibe-devpath.service, WS-N task 1) and, with METRIC_PREFIX=openvibe_toolsjob and its own
+# DEVPATH_RESULT, the Tools job proof (openvibe-toolsjob.service, WS-L task 4).
+# node_exporter's textfile collector reads /var/lib/prometheus/node-exporter/<prefix>.prom.
+# <prefix>_last_run_timestamp_seconds, <prefix>_last_run_ok,
+# <prefix>_last_success_timestamp_seconds (kept from the previous file when this run failed),
+# <prefix>_steps_failed
# A run that refused to start or wrote no result (exit != 0, no fresh last.json) counts as failed.
# A run that refused to start or wrote no result (exit != 0, no fresh last.json) counts as failed.
set -euo pipefail
set -euo pipefail
RESULT=${DEVPATH_RESULT:-/var/lib/openvibe-devpath/last.json}
RESULT=${DEVPATH_RESULT:-/var/lib/openvibe-devpath/last.json}
+PREFIX=${METRIC_PREFIX:-openvibe_devpath}
+[[ "$PREFIX" =~ ^[a-z_]+$ ]] || { echo "bad METRIC_PREFIX" >&2; exit 1; }
DIR=${TEXTFILE_DIR:-/var/lib/prometheus/node-exporter}
DIR=${TEXTFILE_DIR:-/var/lib/prometheus/node-exporter}
[ -d "$DIR" ] || exit 0
[ -d "$DIR" ] || exit 0
-OUT="$DIR/openvibe_devpath.prom"
-node - "$RESULT" "$OUT" "${EXIT_STATUS:-}" <<'JS'
+OUT="$DIR/$PREFIX.prom"
+node - "$RESULT" "$OUT" "${EXIT_STATUS:-}" "$PREFIX" <<'JS'
const fs = require('fs');
const fs = require('fs');
-const [file, out, exitStatus] = process.argv.slice(2);
+const [file, out, exitStatus, p] = process.argv.slice(2);
const now = Math.floor(Date.now() / 1000);
const now = Math.floor(Date.now() / 1000);
let rec = null;
let rec = null;
try { rec = JSON.parse(fs.readFileSync(file, 'utf8')); } catch { rec = null; }
try { rec = JSON.parse(fs.readFileSync(file, 'utf8')); } catch { rec = null; }
const fresh = rec && Date.parse(rec.finished_at) / 1000 > now - 3600;
const fresh = rec && Date.parse(rec.finished_at) / 1000 > now - 3600;
const ok = !!(fresh && rec.ok && (exitStatus === '' || exitStatus === '0'));
const ok = !!(fresh && rec.ok && (exitStatus === '' || exitStatus === '0'));
let lastSuccess = null;
let lastSuccess = null;
-try { const m = /^openvibe_devpath_last_success_timestamp_seconds (\d+)$/m.exec(fs.readFileSync(out, 'utf8')); if (m) lastSuccess = Number(m[1]); } catch { /* first run */ }
+try { const m = new RegExp(`^${p}_last_success_timestamp_seconds (\\d+)$`, 'm').exec(fs.readFileSync(out, 'utf8')); if (m) lastSuccess = Number(m[1]); } catch { /* first run */ }
if (ok) lastSuccess = Math.floor(Date.parse(rec.finished_at) / 1000);
if (ok) lastSuccess = Math.floor(Date.parse(rec.finished_at) / 1000);
const failed = fresh ? rec.steps.filter((s) => !s.ok && !s.skipped).length : 1;
const failed = fresh ? rec.steps.filter((s) => !s.ok && !s.skipped).length : 1;
const lines = [
const lines = [
- '# HELP openvibe_devpath_last_run_timestamp_seconds When the last developer-path run ended.',
- '# TYPE openvibe_devpath_last_run_timestamp_seconds gauge', `openvibe_devpath_last_run_timestamp_seconds ${fresh ? Math.floor(Date.parse(rec.finished_at) / 1000) : now}`,
- '# HELP openvibe_devpath_last_run_ok 1 when every step passed.', '# TYPE openvibe_devpath_last_run_ok gauge', `openvibe_devpath_last_run_ok ${ok ? 1 : 0}`,
- '# HELP openvibe_devpath_steps_failed Steps that failed in the last run.', '# TYPE openvibe_devpath_steps_failed gauge', `openvibe_devpath_steps_failed ${failed}`,
+ `# HELP ${p}_last_run_timestamp_seconds When the last run ended.`,
+ `# TYPE ${p}_last_run_timestamp_seconds gauge`, `${p}_last_run_timestamp_seconds ${fresh ? Math.floor(Date.parse(rec.finished_at) / 1000) : now}`,
+ `# HELP ${p}_last_run_ok 1 when every step passed.`, `# TYPE ${p}_last_run_ok gauge`, `${p}_last_run_ok ${ok ? 1 : 0}`,
+ `# HELP ${p}_steps_failed Steps that failed in the last run.`, `# TYPE ${p}_steps_failed gauge`, `${p}_steps_failed ${failed}`,
];
];
-if (lastSuccess) lines.push('# HELP openvibe_devpath_last_success_timestamp_seconds When the last fully passing run ended.', '# TYPE openvibe_devpath_last_success_timestamp_seconds gauge', `openvibe_devpath_last_success_timestamp_seconds ${lastSuccess}`);
+if (lastSuccess) lines.push(`# HELP ${p}_last_success_timestamp_seconds When the last fully passing run ended.`, `# TYPE ${p}_last_success_timestamp_seconds gauge`, `${p}_last_success_timestamp_seconds ${lastSuccess}`);
fs.writeFileSync(`${out}.tmp`, lines.join('\n') + '\n', { mode: 0o644 });
fs.writeFileSync(`${out}.tmp`, lines.join('\n') + '\n', { mode: 0o644 });
fs.renameSync(`${out}.tmp`, out);
fs.renameSync(`${out}.tmp`, out);
JS
JS