openvibers/openvibe.host Commits

The data role in code (ADR-035, roadmap WS-X2): roles/data/provision.sh installs and configures PostgreSQL 18, PgBouncer, pgBackRest, Valkey and their Prometheus exporters, so any machine can take the role the way this host does. Everything listens on loopback until the private network exists. PostgreSQL is tuned for the shared host (shared_buffers 1 GB, effective_cache_size 4 GB, work_mem 16 MB, 120 connections behind the pooler), uses SCRAM, pg_stat_statements, and WAL archiving through pgBackRest. PgBouncer runs transaction pooling with SCRAM through auth_query (pgbouncer.get_auth, SECURITY DEFINER), so there is no password list to maintain. Valkey uses volatile-lru (only TTL keys are evicted), AOF every second, and ACLs with default off, admin and monitor users, and service users kept in services.acl. pgBackRest keeps an encrypted repository beside ovhost's pruned backup prefix in the same B2 bucket, with a weekly full and daily differential by systemd timer and 14 days of point-in-time recovery. The exporters are bound to 127.0.0.1, because the host firewall is off. add-service.sh gives a service its database, owner role, pooled runtime role (DML only, statement_timeout 15 s, lock_timeout 5 s) and a Valkey user confined to ov:<svc>:*, and writes DATABASE_URL, DATABASE_DIRECT_URL, VALKEY_URL and VALKEY_PREFIX into its env file. Passwords are generated once into /etc/openvibe/data.env (root, 0600) and reach PostgreSQL as SCRAM secrets and Valkey as SHA-256 hashes; nothing prints a secret. .gitignore's data/ rule gains a roles/data/ exception. The Prometheus config generator takes the inventory's exporters. test/data-role.test.js (scripts parse, loopback, SCRAM, transaction pooling, volatile-lru, archiving, no printed secret) and the Prometheus config test. npm test 39/39.

OP OpenVibers committed 28 Sept 2026 · 347930060a · 16 files, +545 −1


Changes

ddiff --git a/.gitignore b/.gitignore
iindex 08cb1fb..4b63cdd 100644
---- a/.gitignore
++++ b/.gitignore
@@@ -11,4 +11,5 @@ host.json
.env
.env.*
data/
++!roles/data/
!.env.example
ddiff --git a/README.md b/README.md
iindex 9a4eade..bc727f8 100644
---- a/README.md
++++ b/README.md
@@@ -36,6 +36,7 @@ Stage A adds a few safety rules of its own:
- Stage A: the host inventory (services, units, env names, ports, probes, drain policy), environment validation, release install and rollback for git-checkout services, readiness and drain orchestration, the release log, certificate inventory, nginx vhost rendering and transactional install, config snapshots, scheduled SQLite backups with retention and encrypted off-host copies, restore drills.
- Stage B: tenant projects (keyed by project id), static sites, immutable content-addressed deploy artifacts, activation and rollback, default and custom domains (DNS TXT verification), per-project quotas, upload/validation logs, the tenant vhosts (`ovhost nginx tenants`).
- Later (Stage C): sandbox profiles, budgets, secret references, outbound policy.
++- Host roles, provisioned from code so any machine can take them: the **data role** (`roles/data/`, ADR-035): PostgreSQL 18, PgBouncer, pgBackRest, Valkey and their exporters, plus `add-service.sh` giving each service its database, roles and Valkey user.
## Does not own
ddiff --git a/roles/data/README.md b/roles/data/README.md
nnew file mode 100644
iindex 0000000..8cf5841
---- /dev/null
++++ b/roles/data/README.md
@@@ -0,0 +1,29 @@
++# The data role (ADR-035, roadmap WS-X2)
++
++PostgreSQL 18, PgBouncer, pgBackRest, Valkey and their exporters, configured from files in this directory,
++so a data server is provisioned the same way as this host. Everything listens on loopback until a private
++network exists (WS-X1).
++
++| Piece | Port | Notes |
++|---|---|---|
++| PostgreSQL 18 | 5432 | `conf.d/openvibe.conf` (tuning, WAL, `pg_stat_statements`), `pg_hba.conf` (SCRAM, loopback) |
++| PgBouncer | 6432 | transaction pooling; SCRAM through `auth_query` (`pgbouncer.get_auth`), no password list to maintain |
++| Valkey | 6379 | `volatile-lru`, AOF every second, ACLs: `default` off, one user per service confined to `ov:<svc>:*` |
++| pgBackRest | — | stanza `openvibe`, encrypted repository beside ovhost's backup prefix in the same B2 bucket, 14 days of PITR; weekly full, daily differential (systemd timers) |
++| Exporters | 9187, 9121, 9127 | PostgreSQL, Valkey, PgBouncer; loopback only |
++
++```bash
++sudo roles/data/provision.sh --check # what would change
++sudo roles/data/provision.sh # apply (idempotent)
++sudo roles/data/add-service.sh wiki # database ov_wiki, roles ov_wiki / ov_wiki_app, Valkey user ov_svc_wiki,
++ # and DATABASE_URL, DATABASE_DIRECT_URL, VALKEY_URL, VALKEY_PREFIX in /etc/openvibe/wiki.env
++```
++
++Generated passwords live only in `/etc/openvibe/data.env` (root, 0600). PostgreSQL receives SCRAM secrets and
++Valkey receives SHA-256 hashes, so no plaintext is ever in SQL, logs, a process list or the scripts' output.
++The one plaintext outside that file is PgBouncer's own auth user in `/etc/pgbouncer/userlist.txt`
++(postgres only). **Escrow `PGBACKREST_CIPHER_PASS`** off the host with the backup key (O8): without it the
++database backups cannot be read.
++
++Applications connect with `DATABASE_URL` (the pooled runtime role: DML only, `statement_timeout` 15 s,
++`lock_timeout` 5 s). Migrations use `DATABASE_DIRECT_URL` (the owner role, session features allowed).
ddiff --git a/roles/data/add-service.sh b/roles/data/add-service.sh
nnew file mode 100755
iindex 0000000..ecb71ce
---- /dev/null
++++ b/roles/data/add-service.sh
@@@ -0,0 +1,90 @@
++#!/usr/bin/env bash
++# roles/data/add-service.sh — give one service its PostgreSQL database and roles and its Valkey user
++# (ADR-035), and write its connection settings into the service's env file.
++#
++# sudo roles/data/add-service.sh <service> [--env-file /etc/openvibe/<service>.env]
++#
++# Per service <svc> (id "-" → "_"):
++# database ov_<id> owned by ov_<id>
++# ov_<id> owner role: migrations and session features, direct on :5432 (DATABASE_DIRECT_URL)
++# ov_<id>_app runtime role through PgBouncer :6432 (DATABASE_URL): DML only, statement_timeout 15 s,
++# lock_timeout 5 s, idle-in-transaction 30 s
++# ov_svc_<id> Valkey user limited to keys and channels under ov:<svc>: (VALKEY_URL, VALKEY_PREFIX)
++# Idempotent: passwords are generated once (/etc/openvibe/data.env) and the env file lines are replaced in
++# place. Nothing is printed but names.
++set -euo pipefail
++umask 077
++HERE=$(cd "$(dirname "$0")" && pwd)
++# shellcheck source=lib.sh
++. "$HERE/lib.sh"
++
++usage() { echo "usage: add-service.sh <service> [--env-file <path>]" >&2; exit 2; }
++[ "$(id -u)" = 0 ] || { echo "add-service.sh: run as root" >&2; exit 1; }
++SVC=${1:-}; [[ "$SVC" =~ ^[a-z][a-z0-9-]{0,30}$ ]] || usage
++ENV_FILE=/etc/openvibe/$SVC.env
++if [ "${2:-}" = "--env-file" ]; then ENV_FILE=${3:?}; elif [ -n "${2:-}" ]; then usage; fi
++ID=${SVC//-/_}; UP=$(echo "$ID" | tr a-z A-Z)
++DB=ov_$ID; OWNER=ov_$ID; APP=ov_${ID}_app; VUSER=ov_svc_$ID; PREFIX="ov:$SVC:"
++
++OWNER_PW=$(secret "PG_${UP}_OWNER_PASSWORD"); APP_PW=$(secret "PG_${UP}_APP_PASSWORD"); V_PW=$(secret "VALKEY_${UP}_PASSWORD")
++OWNER_SCRAM=$(scram "$OWNER_PW"); APP_SCRAM=$(scram "$APP_PW")
++
++# ── PostgreSQL: roles, database, privileges ────────────────────────────────────────────────
++psql_su postgres <<SQL
++DO \$\$ BEGIN
++ IF NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = '$OWNER') THEN CREATE ROLE $OWNER LOGIN; END IF;
++ IF NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = '$APP') THEN CREATE ROLE $APP LOGIN; END IF;
++END \$\$;
++ALTER ROLE $OWNER PASSWORD '$OWNER_SCRAM';
++ALTER ROLE $APP PASSWORD '$APP_SCRAM';
++ALTER ROLE $APP SET statement_timeout = '15s';
++ALTER ROLE $APP SET lock_timeout = '5s';
++ALTER ROLE $APP SET idle_in_transaction_session_timeout = '30s';
++ALTER ROLE $OWNER SET statement_timeout = '0';
++SELECT format('CREATE DATABASE %I OWNER %I', '$DB', '$OWNER') WHERE NOT EXISTS (SELECT 1 FROM pg_database WHERE datname = '$DB') \gexec
++REVOKE ALL ON DATABASE $DB FROM PUBLIC;
++GRANT CONNECT, TEMPORARY ON DATABASE $DB TO $APP;
++GRANT CONNECT ON DATABASE $DB TO ov_monitor;
++SQL
++psql_su "$DB" <<SQL
++ALTER SCHEMA public OWNER TO $OWNER;
++REVOKE ALL ON SCHEMA public FROM PUBLIC;
++GRANT USAGE ON SCHEMA public TO $APP;
++ALTER DEFAULT PRIVILEGES FOR ROLE $OWNER IN SCHEMA public GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO $APP;
++ALTER DEFAULT PRIVILEGES FOR ROLE $OWNER IN SCHEMA public GRANT USAGE, SELECT, UPDATE ON SEQUENCES TO $APP;
++ALTER DEFAULT PRIVILEGES FOR ROLE $OWNER IN SCHEMA public GRANT EXECUTE ON FUNCTIONS TO $APP;
++GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO $APP;
++GRANT USAGE, SELECT, UPDATE ON ALL SEQUENCES IN SCHEMA public TO $APP;
++SQL
++
++# ── Valkey: one user, confined to its prefix ────────────────────────────────────────────────
++TMP=$(mktemp -d); trap 'rm -rf "$TMP"' EXIT
++touch /etc/valkey/services.acl
++{ grep -vE "^user $VUSER " /etc/valkey/services.acl || true
++ echo "user $VUSER on #$(sha256hex "$V_PW") resetkeys ~${PREFIX}* resetchannels &${PREFIX}* +@all -@dangerous"; } > "$TMP/services.acl"
++install -m 640 -o valkey -g valkey "$TMP/services.acl" /etc/valkey/services.acl
++build_valkey_acl "$TMP/users.acl"
++if ! cmp -s "$TMP/users.acl" /etc/valkey/users.acl; then
++ install -m 640 -o valkey -g valkey "$TMP/users.acl" /etc/valkey/users.acl
++ valkey_admin ACL LOAD >/dev/null
++fi
++
++# ── The service's env file: four settings, replaced in place ────────────────────────────────
++[ -f "$ENV_FILE" ] || { install -m 600 /dev/null "$ENV_FILE"; log "created $ENV_FILE"; }
++set_env() {
++ local k=$1 v=$2
++ if grep -qE "^$k=" "$ENV_FILE"; then
++ python3 - "$ENV_FILE" "$k" "$v" <<'PY'
++import sys
++p, k, v = sys.argv[1:4]
++lines = open(p).read().split('\n')
++out = [f'{k}={v}' if l.startswith(k + '=') else l for l in lines]
++open(p, 'w').write('\n'.join(out))
++PY
++ else printf '%s=%s\n' "$k" "$v" >> "$ENV_FILE"; fi
++}
++set_env DATABASE_URL "postgresql://$APP:[email protected]:6432/$DB"
++set_env DATABASE_DIRECT_URL "postgresql://$OWNER:[email protected]:5432/$DB"
++set_env VALKEY_URL "redis://$VUSER:[email protected]:6379/0"
++set_env VALKEY_PREFIX "$PREFIX"
++log "$SVC: database $DB (owner $OWNER, runtime $APP via PgBouncer), Valkey user $VUSER on ${PREFIX}*; DATABASE_URL, DATABASE_DIRECT_URL, VALKEY_URL, VALKEY_PREFIX set in $ENV_FILE (values not shown)"
ddiff --git a/roles/data/files/pg_hba.conf b/roles/data/files/pg_hba.conf
nnew file mode 100644
iindex 0000000..1d27ed5
---- /dev/null
++++ b/roles/data/files/pg_hba.conf
@@@ -0,0 +1,7 @@
++# OpenVibe data role (ADR-035), written by roles/data/provision.sh. Loopback only; the private network
++# (WireGuard, WS-X1) gets its own line when a second host joins.
++# TYPE DATABASE USER ADDRESS METHOD
++local all postgres peer
++local all all scram-sha-256
++host all all 127.0.0.1/32 scram-sha-256
++host all all ::1/128 scram-sha-256
ddiff --git a/roles/data/files/[email protected] b/roles/data/files/[email protected]
nnew file mode 100644
iindex 0000000..1f53ca4
---- /dev/null
++++ b/roles/data/files/[email protected]
@@@ -0,0 +1,11 @@
++[Unit]
++Description=pgBackRest %i backup of the openvibe stanza (ADR-035)
++After=postgresql.service
++
++[Service]
++Type=oneshot
++User=postgres
++Group=postgres
++ExecStart=/usr/bin/pgbackrest --stanza=openvibe --type=%i backup
++Nice=10
++IOSchedulingClass=idle
ddiff --git a/roles/data/files/pgbackrest-diff.timer b/roles/data/files/pgbackrest-diff.timer
nnew file mode 100644
iindex 0000000..288b069
---- /dev/null
++++ b/roles/data/files/pgbackrest-diff.timer
@@@ -0,0 +1,10 @@
++[Unit]
++Description=Daily differential pgBackRest backup (Monday to Saturday 02:15 UTC)
++
++[Timer]
++OnCalendar=Mon..Sat *-*-* 02:15:00 UTC
++Persistent=true
++
++[Install]
++WantedBy=timers.target
ddiff --git a/roles/data/files/pgbackrest-full.timer b/roles/data/files/pgbackrest-full.timer
nnew file mode 100644
iindex 0000000..59884d5
---- /dev/null
++++ b/roles/data/files/pgbackrest-full.timer
@@@ -0,0 +1,10 @@
++[Unit]
++Description=Weekly full pgBackRest backup (Sunday 02:15 UTC; the host backups run at 03:30)
++
++[Timer]
++OnCalendar=Sun *-*-* 02:15:00 UTC
++Persistent=true
++
++[Install]
++WantedBy=timers.target
ddiff --git a/roles/data/files/pgbouncer.ini b/roles/data/files/pgbouncer.ini
nnew file mode 100644
iindex 0000000..5796d5c
---- /dev/null
++++ b/roles/data/files/pgbouncer.ini
@@@ -0,0 +1,27 @@
++; OpenVibe data role (ADR-035): PgBouncer in transaction-pooling mode (ADR-007 2026-09-24 rule 1).
++; Applications use DATABASE_URL (port 6432); migrations use DATABASE_DIRECT_URL (port 5432).
++[databases]
++* = host=127.0.0.1 port=5432
++
++[pgbouncer]
++listen_addr = 127.0.0.1
++listen_port = 6432
++unix_socket_dir = /var/run/postgresql
++auth_type = scram-sha-256
++auth_file = /etc/pgbouncer/userlist.txt
++auth_user = pgbouncer_auth
++auth_dbname = postgres
++auth_query = SELECT usename, passwd FROM pgbouncer.get_auth($1)
++pool_mode = transaction
++max_client_conn = 2000
++default_pool_size = 10
++min_pool_size = 0
++reserve_pool_size = 2
++max_db_connections = 20
++server_idle_timeout = 300
++query_wait_timeout = 30
++ignore_startup_parameters = extra_float_digits,options,application_name
++stats_users = ov_monitor
++admin_users = pgbouncer_auth
++logfile = /var/log/postgresql/pgbouncer.log
++pidfile = /var/run/postgresql/pgbouncer.pid
ddiff --git a/roles/data/files/postgresql-openvibe.conf b/roles/data/files/postgresql-openvibe.conf
nnew file mode 100644
iindex 0000000..81ffa8e
---- /dev/null
++++ b/roles/data/files/postgresql-openvibe.conf
@@@ -0,0 +1,29 @@
++# OpenVibe data role (ADR-035): PostgreSQL 18 settings, installed as conf.d/openvibe.conf by roles/data/provision.sh.
++# Sized for the shared host (4 vCPU, 15 GB RAM) until the data role has its own machine; revisit then.
++listen_addresses = '127.0.0.1' # the private network address is added when a second host joins (WS-X1)
++port = 5432
++max_connections = 120 # applications connect through PgBouncer (transaction pooling)
++password_encryption = scram-sha-256
++
++shared_buffers = 1GB
++effective_cache_size = 4GB
++work_mem = 16MB
++maintenance_work_mem = 256MB
++random_page_cost = 1.1 # SSD
++effective_io_concurrency = 200
++
++wal_level = replica
++max_wal_senders = 5
++max_wal_size = 2GB
++min_wal_size = 256MB
++checkpoint_completion_target = 0.9
++archive_mode = on
++archive_timeout = 300 # at most 5 minutes of WAL outside the repository
++# archive_command is set by provision.sh in conf.d/openvibe-archive.conf (pgBackRest when a repository is configured)
++
++shared_preload_libraries = 'pg_stat_statements'
++pg_stat_statements.track = top
++log_min_duration_statement = 500ms
++log_line_prefix = '%m [%p] %q%u@%d '
++log_lock_waits = on
++idle_in_transaction_session_timeout = 60s
ddiff --git a/roles/data/files/valkey-openvibe.conf b/roles/data/files/valkey-openvibe.conf
nnew file mode 100644
iindex 0000000..3a4767b
---- /dev/null
++++ b/roles/data/files/valkey-openvibe.conf
@@@ -0,0 +1,10 @@
++# OpenVibe data role (ADR-035): Valkey is shared and never authoritative (ADR-007 2026-09-24 rule 4).
++# Included from /etc/valkey/valkey.conf by roles/data/provision.sh.
++bind 127.0.0.1 -::1
++port 6379
++protected-mode yes
++maxmemory 768mb
++maxmemory-policy volatile-lru # only keys with a TTL (caches) are evicted; queues and counters are not
++appendonly yes
++appendfsync everysec
++aclfile /etc/valkey/users.acl
ddiff --git a/roles/data/lib.sh b/roles/data/lib.sh
nnew file mode 100644
iindex 0000000..c8aaff2
---- /dev/null
++++ b/roles/data/lib.sh
@@@ -0,0 +1,65 @@
++# roles/data/lib.sh — helpers shared by provision.sh and add-service.sh (ADR-035). Sourced, never run.
++# No function here prints a secret.
++
++PGV=18
++PGCONF=/etc/postgresql/$PGV/main
++PGDATA=/var/lib/postgresql/$PGV/main
++SECRETS=/etc/openvibe/data.env
++BACKUP_ENV=/etc/openvibe/backup.env
++CHECK=${CHECK:-0}
++
++log() { echo "[data] $*"; }
++
++# install_file SRC DEST MODE OWNER:GROUP → true when DEST changed
++install_file() {
++ if [ -f "$2" ] && cmp -s "$1" "$2"; then return 1; fi
++ if [ "$CHECK" = 1 ]; then log "would install $2"; return 1; fi
++ install -D -m "$3" -o "${4%%:*}" -g "${4##*:}" "$1" "$2"
++ log "installed $2"
++ return 0
++}
++
++# secret NAME → the value, generated once into $SECRETS (root, 0600)
++secret() {
++ mkdir -p /etc/openvibe && touch "$SECRETS" && chmod 600 "$SECRETS"
++ local v
++ v=$(grep -E "^$1=" "$SECRETS" | head -1 | cut -d= -f2- || true)
++ if [ -z "$v" ]; then v=$(openssl rand -hex 24); printf '%s=%s\n' "$1" "$v" >> "$SECRETS"; fi
++ printf '%s' "$v"
++}
++
++# scram PASSWORD → a PostgreSQL SCRAM-SHA-256 secret (RFC 5802/7677), so the plaintext never reaches SQL
++scram() {
++ python3 - "$1" <<'PY'
++import base64, hashlib, hmac, os, sys
++pw = sys.argv[1].encode(); salt = os.urandom(16); it = 4096
++salted = hashlib.pbkdf2_hmac('sha256', pw, salt, it)
++ck = hmac.new(salted, b'Client Key', 'sha256').digest(); sk = hmac.new(salted, b'Server Key', 'sha256').digest()
++b = lambda x: base64.b64encode(x).decode()
++print(f"SCRAM-SHA-256${it}:{b(salt)}${b(hashlib.sha256(ck).digest())}:{b(sk)}", end='')
++PY
++}
++
++sha256hex() { printf '%s' "$1" | sha256sum | cut -d' ' -f1; }
++
++psql_su() { runuser -u postgres -- psql -X -q -v ON_ERROR_STOP=1 -d "${1:-postgres}"; }
++
++# build_valkey_acl OUT → users.acl content: the role's users, then one line per service from services.acl.
++# ACL files take no comments, so the service users live in /etc/valkey/services.acl.
++build_valkey_acl() {
++ local admin mon
++ admin=$(secret VALKEY_ADMIN_PASSWORD); mon=$(secret VALKEY_MONITOR_PASSWORD)
++ touch /etc/valkey/services.acl && chown valkey:valkey /etc/valkey/services.acl && chmod 640 /etc/valkey/services.acl
++ {
++ echo "user default off"
++ echo "user ov_admin on #$(sha256hex "$admin") ~* &* +@all"
++ echo "user ov_monitor on #$(sha256hex "$mon") -@all +info +ping +config|get +client|list +slowlog +latency +dbsize +memory|stats +select"
++ grep -E '^user ov_svc_' /etc/valkey/services.acl || true
++ } > "$1"
++}
++
++# valkey_admin ARGS… → valkey-cli as ov_admin (the password goes through the environment, not argv)
++valkey_admin() {
++ local pw; pw=$(secret VALKEY_ADMIN_PASSWORD)
++ VALKEYCLI_AUTH="$pw" REDISCLI_AUTH="$pw" valkey-cli --no-auth-warning --user ov_admin "$@"
++}
ddiff --git a/roles/data/provision.sh b/roles/data/provision.sh
nnew file mode 100755
iindex 0000000..2015b86
---- /dev/null
++++ b/roles/data/provision.sh
@@@ -0,0 +1,180 @@
++#!/usr/bin/env bash
++# roles/data/provision.sh — install and configure the OpenVibe data role on this host (ADR-035, roadmap WS-X2):
++# PostgreSQL 18, PgBouncer (transaction pooling), pgBackRest (point-in-time recovery to the encrypted B2
++# repository), Valkey (shared, never authoritative) and their Prometheus exporters, all on loopback.
++#
++# sudo roles/data/provision.sh apply roles/data/files/ and create what is missing
++# sudo roles/data/provision.sh --check report what would change, change nothing
++#
++# Idempotent: a second run changes nothing unless files/ changed. Generated passwords live only in
++# /etc/openvibe/data.env (root, 0600); they reach PostgreSQL as SCRAM secrets and Valkey as SHA-256
++# hashes, so no plaintext appears in a log, a process list or this script's output. Per-service
++# databases and users come from roles/data/add-service.sh.
++set -euo pipefail
++umask 077
++
++HERE=$(cd "$(dirname "$0")" && pwd)
++F=$HERE/files
++CHECK=0
++[ "${1:-}" = "--check" ] && CHECK=1
++# shellcheck source=lib.sh
++. "$HERE/lib.sh"
++
++[ "$(id -u)" = 0 ] || { echo "provision.sh: run as root" >&2; exit 1; }
++RESTART_PG=0; RELOAD_PG=0; RESTART_BOUNCER=0; RESTART_VALKEY=0; RELOAD_UNITS=0; RESTART_EXPORTERS=0
++
++# ── 1. Packages ───────────────────────────────────────────────────────────────────────────────
++PKGS=(postgresql-$PGV pgbouncer pgbackrest valkey-server prometheus-postgres-exporter prometheus-redis-exporter prometheus-pgbouncer-exporter)
++MISSING=()
++for p in "${PKGS[@]}"; do dpkg -s "$p" >/dev/null 2>&1 || MISSING+=("$p"); done
++if [ ${#MISSING[@]} -gt 0 ]; then
++ if [ "$CHECK" = 1 ]; then log "would install packages: ${MISSING[*]}"
++ else
++ log "installing ${MISSING[*]}"
++ DEBIAN_FRONTEND=noninteractive apt-get install -y -q "${MISSING[@]}" >/dev/null
++ # Exporters start listening on every interface when installed; stop them until they are bound to loopback below.
++ for u in prometheus-postgres-exporter prometheus-redis-exporter prometheus-pgbouncer-exporter; do systemctl stop "$u" 2>/dev/null || true; done
++ fi
++fi
++[ "$CHECK" = 1 ] && [ ${#MISSING[@]} -gt 0 ] && { log "check: packages missing, stopping here"; exit 0; }
++
++# ── 2. PostgreSQL ────────────────────────────────────────────────────────────────────────────
++install_file "$F/postgresql-openvibe.conf" "$PGCONF/conf.d/openvibe.conf" 644 postgres:postgres && RESTART_PG=1
++install_file "$F/pg_hba.conf" "$PGCONF/pg_hba.conf" 640 postgres:postgres && RELOAD_PG=1
++
++# Archiving: pgBackRest when the off-site repository is configured, else nothing (and say so).
++TMP=$(mktemp -d); trap 'rm -rf "$TMP"' EXIT
++HAVE_REPO=0
++if [ -f "$BACKUP_ENV" ] && grep -q '^BACKUP_S3_BUCKET=' "$BACKUP_ENV"; then HAVE_REPO=1; fi
++if [ "$HAVE_REPO" = 1 ]; then
++ echo "archive_command = 'pgbackrest --stanza=openvibe archive-push %p'" > "$TMP/archive.conf"
++else
++ echo "archive_command = '/bin/true' # no backup repository configured ($BACKUP_ENV): WAL is not archived" > "$TMP/archive.conf"
++ log "WARNING: $BACKUP_ENV has no BACKUP_S3_BUCKET; WAL archiving is off until it does"
++fi
++install_file "$TMP/archive.conf" "$PGCONF/conf.d/openvibe-archive.conf" 644 postgres:postgres && RELOAD_PG=1
++
++# ── 3. pgBackRest (the encrypted repository sits beside, never under, ovhost's pruned backup prefix) ──
++if [ "$HAVE_REPO" = 1 ]; then
++ CIPHER=$(secret PGBACKREST_CIPHER_PASS)
++ (
++ set -a; . "$BACKUP_ENV"; set +a
++ endpoint=${BACKUP_S3_ENDPOINT#https://}; endpoint=${endpoint#http://}; endpoint=${endpoint%/}
++ region=${BACKUP_S3_REGION:-us-east-1}
++ cat > "$TMP/pgbackrest.conf" <<CONF
++# Written by OpenVibe.Host roles/data/provision.sh (ADR-035). Root and postgres only.
++[global]
++repo1-type=s3
++repo1-s3-endpoint=$endpoint
++repo1-s3-bucket=$BACKUP_S3_BUCKET
++repo1-s3-region=$region
++repo1-s3-key=$BACKUP_S3_KEY_ID
++repo1-s3-key-secret=$BACKUP_S3_SECRET
++repo1-s3-uri-style=path
++repo1-path=/openvibe-pgbackrest
++repo1-cipher-type=aes-256-cbc
++repo1-cipher-pass=$CIPHER
++repo1-retention-full=2
++repo1-retention-full-type=count
++process-max=2
++start-fast=y
++compress-type=zst
++log-level-console=warn
++log-level-file=info
++
++[openvibe]
++pg1-path=$PGDATA
++CONF
++ )
++ mkdir -p -m 755 /etc/pgbackrest # umask 077 would make it 700, and postgres must read the file inside
++ install_file "$TMP/pgbackrest.conf" /etc/pgbackrest/pgbackrest.conf 640 root:postgres || true
++ for u in [email protected] pgbackrest-full.timer pgbackrest-diff.timer; do
++ install_file "$F/$u" "/etc/systemd/system/$u" 644 root:root && RELOAD_UNITS=1
++ done
++fi
++
++# ── 4. Valkey ────────────────────────────────────────────────────────────────────────────────
++install_file "$F/valkey-openvibe.conf" /etc/valkey/openvibe.conf 640 valkey:valkey && RESTART_VALKEY=1
++if ! grep -qx 'include /etc/valkey/openvibe.conf' /etc/valkey/valkey.conf; then
++ if [ "$CHECK" = 1 ]; then log "would include openvibe.conf in /etc/valkey/valkey.conf"
++ else echo 'include /etc/valkey/openvibe.conf' >> /etc/valkey/valkey.conf; RESTART_VALKEY=1; log "valkey.conf includes openvibe.conf"; fi
++fi
++if [ "$CHECK" = 0 ]; then
++ build_valkey_acl "$TMP/users.acl"
++ install_file "$TMP/users.acl" /etc/valkey/users.acl 640 valkey:valkey && RESTART_VALKEY=1
++fi
++
++# ── 5. Restart or reload what changed, then the roles that need a running server ────────────
++if [ "$CHECK" = 1 ]; then log "check done"; exit 0; fi
++[ "$RELOAD_UNITS" = 1 ] && systemctl daemon-reload
++systemctl enable -q postgresql valkey-server pgbouncer
++if [ "$RESTART_PG" = 1 ]; then log "restarting PostgreSQL"; systemctl restart "postgresql@$PGV-main"; elif [ "$RELOAD_PG" = 1 ]; then systemctl reload "postgresql@$PGV-main"; fi
++systemctl is-active -q "postgresql@$PGV-main" || systemctl start "postgresql@$PGV-main"
++[ "$RESTART_VALKEY" = 1 ] && { log "restarting Valkey"; systemctl restart valkey-server; }
++systemctl is-active -q valkey-server || systemctl start valkey-server
++
++# Roles every data host has: the PgBouncer auth user (and its lookup function) and the monitoring user.
++BOUNCER_PW=$(secret PGBOUNCER_AUTH_PASSWORD); PGMON_PW=$(secret PG_MONITOR_PASSWORD)
++BOUNCER_SCRAM=$(scram "$BOUNCER_PW"); PGMON_SCRAM=$(scram "$PGMON_PW")
++psql_su postgres <<SQL
++DO \$\$ BEGIN
++ IF NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'pgbouncer_auth') THEN CREATE ROLE pgbouncer_auth LOGIN; END IF;
++ IF NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'ov_monitor') THEN CREATE ROLE ov_monitor LOGIN IN ROLE pg_monitor; END IF;
++END \$\$;
++ALTER ROLE pgbouncer_auth PASSWORD '$BOUNCER_SCRAM';
++ALTER ROLE ov_monitor PASSWORD '$PGMON_SCRAM';
++CREATE SCHEMA IF NOT EXISTS pgbouncer AUTHORIZATION postgres;
++CREATE OR REPLACE FUNCTION pgbouncer.get_auth(p_usename TEXT) RETURNS TABLE(usename name, passwd text)
++ LANGUAGE sql SECURITY DEFINER SET search_path = pg_catalog AS
++ \$f\$ SELECT rolname, rolpassword FROM pg_authid WHERE rolname = p_usename AND rolcanlogin AND NOT rolsuper \$f\$;
++REVOKE ALL ON FUNCTION pgbouncer.get_auth(TEXT) FROM PUBLIC;
++GRANT USAGE ON SCHEMA pgbouncer TO pgbouncer_auth;
++GRANT EXECUTE ON FUNCTION pgbouncer.get_auth(TEXT) TO pgbouncer_auth;
++CREATE EXTENSION IF NOT EXISTS pg_stat_statements;
++SQL
++
++# ── 6. PgBouncer (its auth user's password is the one plaintext it must hold: root and postgres only) ──
++printf '"pgbouncer_auth" "%s"\n' "$BOUNCER_PW" > "$TMP/userlist.txt"
++install_file "$F/pgbouncer.ini" /etc/pgbouncer/pgbouncer.ini 640 postgres:postgres && RESTART_BOUNCER=1
++install_file "$TMP/userlist.txt" /etc/pgbouncer/userlist.txt 640 postgres:postgres && RESTART_BOUNCER=1
++if [ "$RESTART_BOUNCER" = 1 ]; then log "restarting PgBouncer"; systemctl restart pgbouncer; fi
++systemctl is-active -q pgbouncer || systemctl start pgbouncer
++
++# ── 7. Exporters, on loopback only (the host firewall is off: never bind them publicly) ─────
++VALKEY_MON_PW=$(secret VALKEY_MONITOR_PASSWORD)
++cat > "$TMP/pg-exporter" <<EOF
++DATA_SOURCE_NAME='postgresql://ov_monitor:[email protected]:5432/postgres?sslmode=disable'
++ARGS='--web.listen-address=127.0.0.1:9187'
++EOF
++cat > "$TMP/redis-exporter" <<EOF
++REDIS_ADDR='redis://127.0.0.1:6379'
++REDIS_USER='ov_monitor'
++REDIS_PASSWORD='$VALKEY_MON_PW'
++ARGS='-web.listen-address 127.0.0.1:9121'
++EOF
++cat > "$TMP/bouncer-exporter" <<EOF
++ARGS='--web.listen-address=127.0.0.1:9127 --pgBouncer.connectionString=postgres://ov_monitor:[email protected]:6432/pgbouncer?sslmode=disable'
++EOF
++install_file "$TMP/pg-exporter" /etc/default/prometheus-postgres-exporter 640 root:prometheus && RESTART_EXPORTERS=1
++install_file "$TMP/redis-exporter" /etc/default/prometheus-redis-exporter 640 root:prometheus && RESTART_EXPORTERS=1
++install_file "$TMP/bouncer-exporter" /etc/default/prometheus-pgbouncer-exporter 640 root:prometheus && RESTART_EXPORTERS=1
++for u in prometheus-postgres-exporter prometheus-redis-exporter prometheus-pgbouncer-exporter; do
++ systemctl enable -q "$u"
++ if [ "$RESTART_EXPORTERS" = 1 ] || ! systemctl is-active -q "$u"; then systemctl restart "$u"; fi
++done
++
++# ── 8. The pgBackRest stanza and its schedule ───────────────────────────────────────────────
++if [ "$HAVE_REPO" = 1 ]; then
++ if ! runuser -u postgres -- pgbackrest --stanza=openvibe info --output=json 2>/dev/null | grep -q '"status":{"code":0'; then
++ log "creating the pgBackRest stanza"
++ runuser -u postgres -- pgbackrest --stanza=openvibe stanza-create
++ runuser -u postgres -- pgbackrest --stanza=openvibe check
++ log "starting the first full backup ([email protected])"
++ systemctl start --no-block [email protected]
++ fi
++ systemctl enable -q --now pgbackrest-full.timer pgbackrest-diff.timer
++fi
++
++log "done: PostgreSQL $PGV :5432, PgBouncer :6432, Valkey :6379 (all loopback); exporters :9187 :9121 :9127"
++[ "$HAVE_REPO" = 1 ] && log "escrow reminder: PGBACKREST_CIPHER_PASS in $SECRETS decrypts the database backups; keep an off-host copy (O8)"
++exit 0
ddiff --git a/scripts/prometheus-config.js b/scripts/prometheus-config.js
iindex cced71b..13ac01a 100644
---- a/scripts/prometheus-config.js
++++ b/scripts/prometheus-config.js
@@@ -8,7 +8,10 @@
* "sites": { …, "metrics": false } nothing to scrape
*
* Every target is 127.0.0.1 (the services answer /metrics only to direct loopback callers), and
-- * node-exporter (host CPU, memory, disk) is scraped on 127.0.0.1:9100. Alert rules are read from
++ * node-exporter (host CPU, memory, disk) is scraped on 127.0.0.1:9100. The inventory's `exporters`
++ * adds infrastructure exporters, for example the data role's (roles/data, ADR-035):
++ *
++ * "exporters": [{ "name": "postgres", "port": 9187 }, { "name": "valkey", "port": 9121 }, { "name": "pgbouncer", "port": 9127 }] Alert rules are read from
* /etc/prometheus/rules/*.yml (deploy/prometheus/openvibe-rules.yml).
*
* node scripts/prometheus-config.js [--inventory /etc/openvibe/host.json] [--interval 30s] > prometheus.yml
@@@ -48,6 +51,10 @@ function render(inventory, { interval = '30s' } = {}) {
' static_configs:',
" - targets: ['127.0.0.1:9100']",
];
++ for (const e of Array.isArray(inventory.exporters) ? inventory.exporters : []) {
++ if (!/^[a-z][a-z0-9_-]{0,31}$/.test(String(e.name || '')) || !Number.isInteger(e.port)) continue;
++ lines.push(` - job_name: ${e.name}`, ' static_configs:', ` - targets: ['127.0.0.1:${e.port}']`, ' labels:', ` exporter: ${e.name}`);
++ }
for (const j of jobs) {
lines.push(` - job_name: ${j.id}`, ` metrics_path: ${j.path}`, ' static_configs:');
for (const t of j.targets) lines.push(` - targets: ['127.0.0.1:${t.port}']`, ' labels:', ` service: ${j.id}`, ` process: ${t.process}`);
ddiff --git a/test/data-role.test.js b/test/data-role.test.js
nnew file mode 100644
iindex 0000000..f013c2b
---- /dev/null
++++ b/test/data-role.test.js
@@@ -0,0 +1,62 @@
++'use strict';
++// roles/data (ADR-035): the scripts parse, and the settings the ADR fixes are the ones installed: loopback
++// only, SCRAM, transaction pooling, volatile-lru, WAL archiving through pgBackRest, exporters on loopback,
++// secrets generated into data.env and never echoed.
++const assert = require('assert');
++const fs = require('fs');
++const path = require('path');
++const { spawnSync } = require('child_process');
++const { test, runTests } = require('./helpers');
++
++const DIR = path.join(__dirname, '..', 'roles', 'data');
++const read = (f) => fs.readFileSync(path.join(DIR, f), 'utf8');
++
++runTests([
++ test('the scripts parse', () => {
++ for (const f of ['lib.sh', 'provision.sh', 'add-service.sh']) {
++ const r = spawnSync('bash', ['-n', path.join(DIR, f)], { encoding: 'utf8' });
++ assert.strictEqual(r.status, 0, `${f}: ${r.stderr}`);
++ }
++ }),
++ test('PostgreSQL, PgBouncer and Valkey listen on loopback, with SCRAM and the ADR-035 settings', () => {
++ const pg = read('files/postgresql-openvibe.conf');
++ assert.match(pg, /^listen_addresses = '127\.0\.0\.1'/m);
++ assert.match(pg, /^password_encryption = scram-sha-256/m);
++ assert.match(pg, /^shared_buffers = 1GB/m);
++ assert.match(pg, /^archive_mode = on/m);
++ assert.match(pg, /^shared_preload_libraries = 'pg_stat_statements'/m);
++ const hba = read('files/pg_hba.conf').split('\n').filter((l) => l && !l.startsWith('#'));
++ assert.ok(hba.every((l) => /peer|scram-sha-256/.test(l)), 'no trust or md5 lines');
++ assert.ok(hba.every((l) => !/0\.0\.0\.0|::\/0/.test(l)), 'no open addresses');
++ const pb = read('files/pgbouncer.ini');
++ assert.match(pb, /^listen_addr = 127\.0\.0\.1/m);
++ assert.match(pb, /^pool_mode = transaction/m);
++ assert.match(pb, /^auth_type = scram-sha-256/m);
++ const vk = read('files/valkey-openvibe.conf');
++ assert.match(vk, /^bind 127\.0\.0\.1 -::1/m);
++ assert.match(vk, /^maxmemory-policy volatile-lru/m);
++ assert.match(vk, /^aclfile /m);
++ }),
++ test('exporters bind loopback, archiving goes through pgBackRest, and secrets are never echoed', () => {
++ const p = read('provision.sh');
++ for (const port of ['9187', '9121', '9127']) assert.match(p, new RegExp(`127\\.0\\.0\\.1:${port}|127\\.0\\.0\\.1 *:?${port}`));
++ assert.match(p, /pgbackrest --stanza=openvibe archive-push %p/);
++ assert.match(p, /repo1-cipher-type=aes-256-cbc/);
++ assert.match(p, /repo1-path=\/openvibe-pgbackrest/, 'beside, never under, the pruned backup prefix');
++ const lib = read('lib.sh');
++ assert.match(lib, /SCRAM-SHA-256\$\{it\}/, 'passwords reach SQL as SCRAM secrets');
++ // A secret may be hashed (sha256hex, scram) or written into a file by redirection; never printed to the output.
++ const SECRET_VAR = /\$\{?[A-Z_]*(PW|PASSWORD|CIPHER|SECRET)\b/;
++ for (const f of ['lib.sh', 'provision.sh', 'add-service.sh']) {
++ read(f).split('\n').forEach((line, i) => {
++ if (!/\b(echo|printf)\b/.test(line) || !SECRET_VAR.test(line)) return;
++ const stripped = line.replace(/\$\((sha256hex|scram) "\$\{?[A-Z_]+\}?"\)/g, '');
++ const toFile = /(>>?)\s*"?\$?[\w{}\/.\-"$]+"?\s*;?\s*(\}|$)/.test(line) || /\}\s*>\s*"/.test(line);
++ assert.ok(!SECRET_VAR.test(stripped) || toFile, `${f}:${i + 1} prints a secret: ${line.trim()}`);
++ });
++ }
++ const add = read('add-service.sh');
++ assert.match(add, /ALTER ROLE \$APP SET statement_timeout = '15s'/);
++ assert.match(add, /~\$\{PREFIX\}\*/, 'the Valkey user is confined to its prefix');
++ }),
++]);
ddiff --git a/test/prometheus-config.test.js b/test/prometheus-config.test.js
iindex fe4a643..2350543 100644
---- a/test/prometheus-config.test.js
++++ b/test/prometheus-config.test.js
@@@ -18,4 +18,9 @@ assert.ok(out.includes(" - targets: ['127.0.0.1:4012']\n labels:\n
assert.ok(!out.includes('bad name') && !out.includes(':1\''), 'malformed extras are skipped');
assert.ok(!out.includes('job_name: sites') && !out.includes('job_name: noport'), 'opted out or nothing to scrape');
assert.ok(!/0\.0\.0\.0|localhost/.test(out), 'loopback addresses only');
++// The data role's exporters (roles/data, ADR-035), from the inventory's `exporters`; malformed entries are skipped.
++const withExporters = render({ services: {}, exporters: [{ name: 'postgres', port: 9187 }, { name: 'valkey', port: 9121 }, { name: 'Bad', port: 9 }, { name: 'x' }] });
++assert.ok(withExporters.includes(" - job_name: postgres\n static_configs:\n - targets: ['127.0.0.1:9187']\n labels:\n exporter: postgres\n"));
++assert.ok(withExporters.includes("targets: ['127.0.0.1:9121']"));
++assert.ok(!withExporters.includes('Bad') && !withExporters.includes('job_name: x'), 'malformed exporters are skipped');
console.log('prometheus config: all checks passed');