openvibers/openvibe.host Commits
Host: Mail inventory with Stalwart as an auxiliary unit (#324)
Changes
@@@ -288,7 +288,7 @@ The sandbox blocks writes outside the drill directory and addresses beyond loopb### Inventory--`host.example.json` describes every service on the host. It records the checkout, owner/`runAs`, units and socket unit, `unitSources` (repo unit files), env file, `env.required` (names, or `"from-example"`), port, loopback `ready` URL (plus headers, since Tools needs `Host: openvibe.tools`), `packages` (`"apps/*"` for Tools), install command, `build` hooks, `noRestartPaths`, the `protected` probe (`http-json-count` — a numeric `field`, plus any present `addFields` —, `postgresql-count`, or `sum` of several: Live streams, Media recordings, chunk sessions and uploads, Events SSE connections, Chat sockets, Games players online, OpenRestream ingest sessions, Tools running jobs), the `drain` policy, `databases` (each a PostgreSQL database `{ "name", "engine": "postgresql", "database": "ov_<name>" }`; [docs/db-inventory.md](docs/db-inventory.md)), `backupOnChange`, `nginx`, `layout` (`release` for a `releases/<id>` + `current` service using the `release-layout` strategy) and `workerUnits`. `workerUnits` are units, or templates such as OpenRestream's `[email protected]`, that belong to the service but that ovhost **never starts, stops or restarts**: they drain and exit on their own. `status` and `validate` list their instances read-only, `validate` warns when none runs, and the inventory refuses a worker unit, or an instance of one, in `units`. An `addFields` entry that the service does not report at all is read as zero (an older service), while one it reports as `null` is read as unknown: the service could not answer it, so the restart is held. The real file is `/etc/openvibe/host.json` and is **never committed**. When `ovhost` runs as root it refuses an inventory that is not root-owned or that anyone else can write, and it rejects relative paths, non-loopback probe URLs, non-SELECT probe queries and vhost names that contain a path.++`host.example.json` describes every service on the host. It records the checkout, owner/`runAs`, units and socket unit, `unitSources` (repo unit files), env file, `env.required` (names, or `"from-example"`), port, loopback `ready` URL (plus headers, since Tools needs `Host: openvibe.tools`), `packages` (`"apps/*"` for Tools), install command, `build` hooks, `noRestartPaths`, the `protected` probe (`http-json-count` — a numeric `field`, plus any present `addFields` —, `postgresql-count`, or `sum` of several: Live streams, Media recordings, chunk sessions and uploads, Events SSE connections, Chat sockets, Games players online, OpenRestream ingest sessions, Tools running jobs), the `drain` policy, `databases` (each a PostgreSQL database `{ "name", "engine": "postgresql", "database": "ov_<name>" }`; [docs/db-inventory.md](docs/db-inventory.md)), `backupOnChange`, `nginx`, `layout` (`release` for a `releases/<id>` + `current` service using the `release-layout` strategy) and `workerUnits`. `workerUnits` are units, or templates such as OpenRestream's `[email protected]`, that belong to the service but that ovhost **never starts, stops or restarts**: they drain and exit on their own. `status` and `validate` list their instances read-only, `validate` warns when none runs, and the inventory refuses a worker unit, or an instance of one, in `units`. `auxiliaryUnits` are service units a service depends on but does not run in its own process: Mail's `openvibe-stalwart.service`, installed from `unitSources` (which may also carry its `.slice`). A deploy restarts one only when a path in its `restartOn` list changed, and a rollback never does. `objects` directories are archived by `backup`; when a store compresses or encrypts its blobs, declare `"verify": false` so a drill checks the archive but not each file's sha256. An `addFields` entry that the service does not report at all is read as zero (an older service), while one it reports as `null` is read as unknown: the service could not answer it, so the restart is held. The real file is `/etc/openvibe/host.json` and is **never committed**. When `ovhost` runs as root it refuses an inventory that is not root-owned or that anyone else can write, and it rejects relative paths, non-loopback probe URLs, non-SELECT probe queries and vhost names that contain a path.**Lifecycle** (roadmap WS-P task 1, `lib/lifecycle.js`). Each service's lifecycle is the `lifecycle` block of its manifest in openvibe-contracts (≥ 0.55.0): liveness, shutdown (signal, `deadlineSeconds`, drains, workers), startupRecovery, rollback, contracts and leases. A service entry may carry its own `lifecycle` block, which replaces the manifest's on this host, and `validate --manifest <file>` reads another manifest file. `ovhost validate` fails when no block is found or a field is missing, naming the service and the field (`live: lifecycle.shutdown.deadlineSeconds is missing`). It also fails when `deadlineSeconds` exceeds a unit's stop timeout (systemd's effective `TimeoutStopUSec`, else `TimeoutStopSec` in the unit source) or the unit's `KillSignal` is another signal, and when the checkout's installed openvibe-contracts is outside `contracts.range`. Worker units that drain longer than their stop timeout are a warning, since ovhost never stops them. This repository still pins openvibe-contracts v0.49.0, which has no lifecycle blocks, so until that pin moves to v0.55.0 or later `validate` reports every service as undeclared.@@@ -11,7 +11,9 @@ Restore drills (a second instance started from a backup and compared with produc## What is backed up--Every inventory database is PostgreSQL. `ovhost backup --all` verifies the cluster's pgBackRest backups and WAL archiver on every run. It makes a per-service `pg_dump -Fc` when the last good logical dump is over seven days old or when `--logical` is given. Declared content-addressed object directories are archived as `.tar.gz` files. Restore drills use the logical dumps and object archives.++Every inventory database is PostgreSQL. `ovhost backup --all` verifies the cluster's pgBackRest backups and WAL archiver on every run. It makes a per-service `pg_dump -Fc` when the last good logical dump is over seven days old or when `--logical` is given. Declared object directories are archived as `.tar.gz` files. Restore drills use the logical dumps and object archives.++++A directory whose files are named by their own sha256 is content-addressed and a drill verifies every blob. Some stores compress or encrypt their blobs (Stalwart's message blobs under `/var/lib/openvibe-stalwart/blobs`), so the bytes do not hash to the file name: declare `"verify": false` in the inventory's `objects` entry. The directory is still archived, uploaded and restored; the drill checks the archive, not each blob.## One run@@@ -13,3 +13,8 @@ Every `services.<id>.databases[]` entry names a database in the host PostgreSQL```The drill creates a scratch database and role, restores the archive, and points both settings at the scratch database. A `postgresql-count` protected-session probe runs a read-only SELECT through the postgres OS user.++++Mail's `ov_mail` is both the Mail service's database and Stalwart's data store; Stalwart's tables++live in the same `public` schema. `sudo ovhost data provision mail` creates the `ov_mail` owner and++`ov_mail_app` runtime roles. Stalwart connects directly as the owner (`STALWART_DB_USER=ov_mail`,++`STALWART_DB_PASSWORD`); Mail's migrations run on `DATABASE_DIRECT_URL`, the same owner on :5432.@@@ -47,6 +47,8 @@ Repository wrappers call `ovhost plan`, `ovhost deploy`, or `ovhost rollback` fonothing restarted (exit 2).7. Restart the units (plus unit files matching `unitsMatch`), wait for readiness (`ready.release`,`ready.allUnits`), and on failure roll back: restore, reinstall, rebuild, restart (exit 3, or 4).++ An auxiliary unit (`auxiliaryUnits`, Mail's Stalwart) is restarted first, and only when one of its++ `restartOn` paths changed; the service itself is not restarted for a Stalwart-only change.8. Record the release; announce it (Sites: one notification per `dist/<domain>/release.json`).### Release layout (release-layout)@@@ -105,6 +107,7 @@ overrides them.| `generated` | tracked paths a build rewrites; restored from git before the merge, never a local change | static-build: `["dist/"]` || `installUnits` | install drifted `unitSources` on every deploy (as `--install-units`) | false || `unitsMatch` | also restart unit files matching this glob (named when the inventory lacks them) | none |++| `auxiliaryUnits` | `[{ unit, restartOn: [paths] }]` units the service depends on but that an ordinary deploy must not restart (Mail's Stalwart). Installed from `unitSources`, listed by `status`/`validate`; restarted before the service only when a `restartOn` path changed. A rollback never restarts them | `[]` || `announce.releaseFiles` | one notification per `<dir>/*/release.json` instead of one for the service | static-build: `dist/*/release.json` || `install.lockfile` | the lockfile that decides an install | `package-lock.json` (pnpm-build: `pnpm-lock.yaml`) || `install.workspace` | one install at the root for a workspace | pnpm-build: true |@@@ -34,3 +34,18 @@ Why it is not switched on without the owner:- the provider's edge firewall may already filter some of these ports.Enable it at a quiet time, with a second SSH session open, then run the browser check and a WebRTC publish and view.++++## OpenVibe.Mail (M1, 2026-11-07)++++Inbound **TCP 25 only**. Stalwart M1 is receive-only: the rendered plan removes its submission++(587), SMTPS (465), POP3, Sieve and HTTPS listeners before they open. JMAP/management (8080) and++IMAP (143) stay on loopback; the Mail API (5250) is loopback behind nginx.++++When the default-deny proposal above is enabled, add only:++++```++sudo ufw allow 25/tcp++```++++Do **not** add 587 or 465. If `sudo ss -ltnp | grep -E ':(587|465|993|995|4190)\b'` names Stalwart,++stop it and re-apply `deploy/stalwart/apply.js` before opening anything.@@@ -4003,6 +4003,146 @@"countsTolerance": 5,"_notes": "The restored copy boots with no service credentials (no tokens, no outbox publishing or subscriptions), Valkey on a closed port and every outbound service URL on a closed port; counts and readiness."}++ },++ "mail": {++ "_note": "OpenVibe.Mail (M1): receive-only Stalwart on inbound TCP 25, masks, burners and agent inboxes; database ov_mail; public door openvibe.email. openvibe-stalwart.service and its slice are auxiliary: installed from this checkout and listed by status/validate, but a Mail deploy restarts them only when deploy/stalwart/config.toml or one of their unit files changes. The binary is installed by hand from scripts/fetch-stalwart.js (pinned sha256) — ovhost never builds or downloads it. Message blobs at /var/lib/openvibe-stalwart/blobs are compressed and named by content hash, so they are archived by backup with verify:false and not sha256-checked. Remove openvibe.email from /etc/nginx/sites-enabled/parked-domains.conf before the vhost can win the name. Ports 587 and 465 stay closed.",++ "repo": "/opt/openvibe.mail",++ "units": [++ "openvibe-mail.service"++ ],++ "auxiliaryUnits": [++ {++ "unit": "openvibe-stalwart.service",++ "restartOn": [++ "deploy/systemd/openvibe-stalwart.service",++ "deploy/systemd/openvibe-stalwart.slice"++ ],++ "applyOn": [++ "deploy/stalwart/"++ ],++ "applyHint": "render and apply it as OpenVibe.Mail docs/INSTALL.md says (render.js, then apply.js with the plan); a restart alone keeps the old configuration"++ }++ ],++ "unitSources": {++ "openvibe-mail.service": "deploy/systemd/openvibe-mail.service",++ "openvibe-stalwart.service": "deploy/systemd/openvibe-stalwart.service",++ "openvibe-stalwart.slice": "deploy/systemd/openvibe-stalwart.slice"++ },++ "installUnits": true,++ "enableUnits": true,++ "envFile": "/etc/openvibe/mail.env",++ "env": {++ "required": [++ "NODE_ENV",++ "PORT",++ "HOST",++ "BASE_URL",++ "TRUST_PROXY",++ "DATABASE_URL",++ "DATABASE_DIRECT_URL",++ "VALKEY_URL",++ "VALKEY_PREFIX",++ "OV_NETWORK_URL",++ "OV_NETWORK_INTERNAL_URL",++ "OV_OAUTH_CLIENT_ID",++ "OV_OAUTH_CLIENT_SECRET",++ "MAIL_BASE_DOMAIN",++ "MAIL_MASK_DOMAIN",++ "BURNER_DOMAINS",++ "STALWART_URL",++ "STALWART_ADMIN_USER",++ "STALWART_ADMIN_PASSWORD"++ ]++ },++ "port": 5250,++ "health": {++ "url": "http://127.0.0.1:5250/api/health"++ },++ "ready": {++ "url": "http://127.0.0.1:5250/api/ready",++ "timeoutSeconds": 30++ },++ "noRestartPaths": [++ "docs/",++ "test/",++ "**/*.md",++ "deploy/stalwart/",++ "deploy/systemd/openvibe-stalwart.service",++ "deploy/systemd/openvibe-stalwart.slice"++ ],++ "databases": [++ {++ "name": "mail",++ "engine": "postgresql",++ "database": "ov_mail",++ "role": "ov_mail"++ }++ ],++ "objects": [++ {++ "name": "stalwart-blobs",++ "path": "/var/lib/openvibe-stalwart/blobs",++ "verify": false++ }++ ],++ "drill": {++ "supported": false,++ "reason": "M1 has no drill switch: readiness depends on the live Stalwart directory and the process writes the directory view at boot, so a second instance is not side-effect free."++ },++ "nginx": {++ "vhost": "openvibe.email.conf",++ "repoVhost": "deploy/nginx/openvibe.email.conf",++ "installOnDeploy": true++ },++ "lifecycle": {++ "liveness": {++ "endpoint": "/api/health",++ "means": "the process answers HTTP; nothing else is checked"++ },++ "shutdown": {++ "signal": "SIGTERM",++ "deadlineSeconds": 10,++ "drains": [++ "the HTTP server closes",++ "the purge job stops",++ "the database connection closes"++ ]++ },++ "startupRecovery": {++ "resumes": [++ {++ "kind": "jobs",++ "what": "due mask expiries and burner purges run on the next interval"++ },++ {++ "kind": "outbox",++ "what": "unsent mail.received and mail.mask.created events stay in the outbox until the relay starts"++ }++ ]++ },++ "rollback": {++ "conditions": [++ "automatic: not ready within ready.timeoutSeconds"++ ],++ "window": "while ovhost waits for readiness",++ "blockers": {++ "none": "the service's migrations only add; Stalwart's tables live in the same database and are not touched by a Mail release"++ }++ },++ "contracts": {++ "range": ">=0.329.0 <1.0.0"++ },++ "leases": {++ "claims": [++ {++ "what": "the purge job lease",++ "holder": "the one mail process",++ "expires": "MAIL_PURGE_INTERVAL_MS (60 s)",++ "fencing": "a conditional UPDATE on mail_settings.updated_at"++ }++ ]++ }++ }}}}@@@ -270,9 +270,9 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc// The inventory as ovhost reads it (defaults applied). It holds no secret values.const ids = pos.slice(1);const list = ids.length ? ids.map((id) => lib.inventory.service(inv, id)) : Object.values(inv.services);-- const rows = list.map((s) => ({ id: s.id, managed: s.managed, repo: s.repo, owner: s.owner, runAs: s.runAs, branch: s.branch, units: s.units, socketUnit: s.socketUnit || null, workerUnits: s.workerUnits, port: s.port || null, ready: s.ready ? s.ready.url : null, protected: s.protected ? s.protected.kind : null, drain: s.drain ? s.drain.policy : null, layout: s.layout }));++ const rows = list.map((s) => ({ id: s.id, managed: s.managed, repo: s.repo, owner: s.owner, runAs: s.runAs, branch: s.branch, units: s.units, socketUnit: s.socketUnit || null, workerUnits: s.workerUnits, auxiliaryUnits: s.auxiliaryUnits.map((a) => a.unit), port: s.port || null, ready: s.ready ? s.ready.url : null, protected: s.protected ? s.protected.kind : null, drain: s.drain ? s.drain.policy : null, layout: s.layout }));if (json) { emit(ids.length === 1 ? rows[0] : rows); return 0; }-- for (const r of rows) out(`${r.id.padEnd(10)} ${r.repo} units: ${r.units.join(' ') || '-'}${r.socketUnit ? ` socket: ${r.socketUnit}` : ''}${r.workerUnits.length ? ` workers: ${r.workerUnits.join(' ')}` : ''}${r.protected ? ` protected: ${r.protected}` : ''}${r.managed ? '' : ' [unmanaged]'}`);++ for (const r of rows) out(`${r.id.padEnd(10)} ${r.repo} units: ${r.units.join(' ') || '-'}${r.socketUnit ? ` socket: ${r.socketUnit}` : ''}${r.workerUnits.length ? ` workers: ${r.workerUnits.join(' ')}` : ''}${r.auxiliaryUnits.length ? ` auxiliary: ${r.auxiliaryUnits.join(' ')}` : ''}${r.protected ? ` protected: ${r.protected}` : ''}${r.managed ? '' : ' [unmanaged]'}`);return 0;}case 'env-names': {@@@ -301,6 +301,9 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => procif (json) { emit(p); return 0; }out(`${p.service}: ${short(p.from)} → ${short(p.to)}${p.upToDate ? ' (up to date)' : ''} on ${p.branch}${p.branch !== p.expectedBranch ? ` (EXPECTED ${p.expectedBranch})` : ''}`);out(` strategy ${p.strategy || 'none'}${p.layout === 'release' ? ` (release layout, keep ${p.keep})` : ''}`);++ out(` repo ${p.repo}`);++ if (p.envFile) out(` env ${p.envFile}`);++ if (p.databases && p.databases.length) out(` databases ${p.databases.join(', ')}`);if (p.layout === 'release') out(` release ${p.fromRelease} → ${p.upToDate ? p.fromRelease : p.reuse ? `${p.toRelease} (exists; reused)` : `${p.toRelease} (new, prepared while ${p.fromRelease} serves)`}`);for (const c of p.commits.slice(0, 30)) out(` ${c}`);out(` files changed ${p.changedFiles} (${p.codeChanged} outside the no-restart paths)`);@@@ -312,6 +315,7 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => procif (p.build && p.build.length) out(` build ${p.build.join(' && ')}`);if (p.preflight && p.preflight.length) out(` preflight ${p.preflight.join('; ')}`);out(` restart ${p.restartNeeded ? `yes: ${p.units.join(', ')}${p.socketUnit ? ` (socket ${p.socketUnit} stays up)` : ''}` : p.layout === 'release' && !p.upToDate ? 'no — current switches, the process keeps running' : 'no'}`);++ if (p.auxiliaryUnits && p.auxiliaryUnits.length) out(` auxiliary ${p.auxiliaryUnits.join(', ')} (installed with the service; not restarted unless its config changes${p.auxRestartNeeded ? ` — restarting now: ${p.auxRestart.join(', ')}` : ''})`);if (p.extraUnits && p.extraUnits.length) out(` units ${p.extraUnits.join(', ')} are not in the inventory but are restarted too (unitsMatch)`);if (p.skippedJobs && p.skippedJobs.length) out(` jobs ${p.skippedJobs.join(', ')} match unitsMatch but are oneshot jobs: not restarted, not waited on`);if (p.socket) out(` socket ${p.socket.held ? `held by systemd on :${p.socket.port}` : p.socket.held === false ? `NOT held by systemd on :${p.socket.port} — a restart rebinds it (stop, restart the socket, start)` : 'unknown'}${p.socket.changed ? '; the socket unit file changed — a restart rebinds it' : ''}`);@@@ -321,7 +325,7 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc}if (p.backupNeeded) out(' backup schema files changed — databases are backed up before the restart');if (p.unitDrift.length) out(` unit files differ from the repo: ${p.unitDrift.map((u) => u.unit).join(', ')}${p.installUnits ? ' (installed by the deploy)' : ' (--install-units)'}`);-- if (p.installVhosts) out(' nginx repo vhosts are installed (nginx -t, reload) on deploy');++ if (p.installVhosts) out(` nginx ${p.nginxVhost ? `${p.nginxVhost.name} from ${p.nginxVhost.source} installed` : 'repo vhosts installed'} (nginx -t, reload) on deploy`);for (const v of p.vhostsRemoved || []) out(` nginx ${v} left the repository; it stays installed (remove by hand)`);if (p.nginxNotInstalled) out(' nginx the repo vhost changed and is NOT installed by a deploy (by hand: nginx -t, reload)');if (p.announceFrom) out(` announce one release notification per ${p.announceFrom}`);@@@ -386,9 +390,10 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => procconst units = r.units.map((u) => `${u.unit.replace(/\.service$/, '')}=${u.active}${u.restarts ? `(${u.restarts} restarts)` : ''}`).join(' ');const sock = r.socket ? ` socket=${r.socket.active}` : '';const workers = r.workers && r.workers.length ? ` workers[${r.workers.map((w) => `${w.unit.replace(/\.service$/, '')}=${w.active}`).join(' ')}]` : '';++ const auxiliary = r.auxiliary && r.auxiliary.length ? ` aux[${r.auxiliary.map((a) => `${a.unit.replace(/\.service$/, '')}=${a.active}`).join(' ')}]` : '';const ready = r.ready ? (r.ready.ok ? 'ready' : `NOT READY (${r.ready.status || r.ready.error})`) : '-';const prot = r.protected ? (r.protected.count == null ? ` ${r.protected.label}=unknown` : ` ${r.protected.label}=${r.protected.count}`) : '';-- out(`${r.service.padEnd(10)} ${short(r.sha)} ${ready.padEnd(12)} ${units || '(no units)'}${sock}${workers}${prot}${r.managed ? '' : ' [unmanaged]'}`);++ out(`${r.service.padEnd(10)} ${short(r.sha)} ${ready.padEnd(12)} ${units || '(no units)'}${sock}${workers}${auxiliary}${prot}${r.managed ? '' : ' [unmanaged]'}`);}return 0;}@@@ -260,7 +260,7 @@ async function resolveBackup(exec, inv, svc, dirArg) {// to extract. A --backup directory has no record; its compressed size is the best we can do.const rec = record && (record.files || []).find((f) => f.file === `${obj.name}.tar.gz`);const sourceBytes = rec && Number.isFinite(rec.sourceBytes) ? rec.sourceBytes : fst.size;-- objects.push({ name: obj.name, file, production: obj.path, sourceBytes });++ objects.push({ name: obj.name, file, production: obj.path, sourceBytes, verify: obj.verify !== false });}return { dir, takenAt, files, objects };}@@@ -468,9 +468,15 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {const entry = { name: obj.name, source: obj.file, copy: dest, blobs: 0, skipped: 0, bytes: 0, mismatches: 0, integrity: null };record.objects.push(entry);log(`restoring object directory ${obj.file} → ${dest} (${svc.runAs})`);-- try { await verifyObjectBlobs(exec, dest, entry); } catch (err) { entry.integrity = `error: ${err.message}`; throw fail('objects', `${obj.name}: ${err.message}`); }-- entry.integrity = 'sha256 ok';-- log(`object directory ${obj.name}: ${entry.blobs} blob(s) sha256 ok${entry.skipped ? `, ${entry.skipped} non-blob file(s) skipped` : ''}`);++ if (obj.verify === false) {++ // The store compresses or encrypts each blob, so a file does not hash to its name.++ entry.integrity = 'archive only (blobs are not content-addressed on disk)';++ log(`object directory ${obj.name}: restored (integrity is by the archive; its blobs are not content-addressed)`);++ } else {++ try { await verifyObjectBlobs(exec, dest, entry); } catch (err) { entry.integrity = `error: ${err.message}`; throw fail('objects', `${obj.name}: ${err.message}`); }++ entry.integrity = 'sha256 ok';++ log(`object directory ${obj.name}: ${entry.blobs} blob(s) sha256 ok${entry.skipped ? `, ${entry.skipped} non-blob file(s) skipped` : ''}`);++ }}const restoredCounts = [];for (const c of d.counts) {@@@ -671,7 +677,9 @@ function markdownRow(r) {for (const db of r.databases) checks.push(`\`pg_restore --list\` (${db.name}) = ${db.integrity === 'pg_restore --list ok' ? 'ok' : cell(db.integrity).slice(0, 80)}`);for (const o of r.objects || []) checks.push(o.integrity === 'sha256 ok'? `object store \`${o.name}\` ${o.blobs} blob(s) sha256 ok`-- : `object store \`${o.name}\`: ${cell(o.integrity).slice(0, 80)}`);++ : o.integrity && o.integrity.startsWith('archive only')++ ? `object store \`${o.name}\`: archive restored (blobs not content-addressed)`++ : `object store \`${o.name}\`: ${cell(o.integrity).slice(0, 80)}`);if (r.ready) checks.push(r.ready.ok ? `drill instance \`${new URL(r.ready.url).pathname}\` ${r.ready.status} after ${r.ready.seconds}s` : `drill instance not ready (${r.ready.status || 'no answer'})`);const same = r.compare.filter((c) => c.match).map((c) => `\`${c.path}\``);const diff = r.compare.filter((c) => !c.match).map((c) => `\`${c.path}\` differs (${cell(c.detail)})`);@@@ -7,11 +7,16 @@ const { git } = require('../git');async function statusOne(ctx, svc) {const { exec } = ctx;-- const row = { service: svc.id, managed: svc.managed, units: [], workers: [], socket: null, sha: null, branch: null, ready: null, protected: null };++ const row = { service: svc.id, managed: svc.managed, units: [], workers: [], auxiliary: [], socket: null, sha: null, branch: null, ready: null, protected: null };for (const u of svc.units) {const st = await systemd.show(exec, u);row.units.push({ unit: u, active: st.active, sub: st.sub, restarts: st.restarts });}++ // Auxiliary units (Mail's Stalwart) belong to the service but are never restarted with it.++ for (const a of svc.auxiliaryUnits) {++ const st = await systemd.show(exec, a.unit);++ row.auxiliary.push({ unit: a.unit, active: st.active, sub: st.sub, restarts: st.restarts });++ }// Worker units are only listed: ovhost never starts, stops or restarts them.for (const w of svc.workerUnits) {try {@@@ -123,6 +123,11 @@ async function validate(ctx, id, opts = {}) {if (!running.length) add('warn', 'units', `no running instance of worker unit ${w}`);for (const i of list) add('info', 'units', `worker ${i.unit} ${i.active}/${i.sub} (ovhost never starts, stops or restarts it)`);}++ for (const a of svc.auxiliaryUnits) {++ const st = await systemd.show(exec, a.unit);++ if (st.load !== 'loaded' || !st.fragmentPath) add('error', 'units', `auxiliary ${a.unit} is not loaded (${st.load})`);++ else add(st.active === 'active' ? 'ok' : 'warn', 'units', `auxiliary ${a.unit} ${st.active}/${st.sub} (restarted with ${svc.id} only when ${a.restartOn.length ? a.restartOn.join(', ') : 'a declared path'} changes)`);++ }if (svc.socketUnit) {const st = await systemd.show(exec, svc.socketUnit);if (st.load !== 'loaded') add('error', 'units', `${svc.socketUnit} is not loaded (${st.load})`);@@@ -16,8 +16,12 @@ class InventoryError extends Error {}const ID_RE = /^[a-z][a-z0-9-]{0,39}$/;const UNIT_RE = /^[A-Za-z0-9@._-]+\.(service|socket)$/;++// unitSources may also carry a slice (Stalwart's memory cap) and drop-ins; only .service units are restarted.++const SOURCE_UNIT_RE = /^[A-Za-z0-9@._-]+\.(service|socket|slice)$/;// A worker unit is a template (`[email protected]`) or a plain service unit.const WORKER_UNIT_RE = /^[A-Za-z0-9@._-]+\.service$/;++// An auxiliary unit is a plain service unit that belongs to a service but is not restarted with it.++const AUX_UNIT_RE = /^[A-Za-z0-9@._-]+\.service$/;const FILE_NAME_RE = /^[A-Za-z0-9._-]+$/;const DRAIN_POLICIES = ['refuse', 'wait', 'report'];const PROBE_KINDS = ['http-json-count', 'postgresql-count', 'sum'];@@@ -144,10 +148,12 @@ function normalisePreflight(raw, where) {}/**-- * `objects`: directories of content-addressed blobs the service owns OUTSIDE its databases, such as-- * OpenVibe.Host's tenant object store (host.example.json: `HOST_STORAGE_DIR`). Each regular file in-- * the directory is named by its own sha256, so `ovhost backup` can archive the tree as one artifact-- * and a drill can check every blob's sha256 after restoring it.++ * `objects`: directories of blobs the service owns OUTSIDE its databases, such as OpenVibe.Host's++ * tenant object store (host.example.json: `HOST_STORAGE_DIR`) or Stalwart's message blobs. When the++ * files are named by their own sha256 (the default), `ovhost backup` archives the tree as one++ * artifact and a drill checks every blob's sha256 after restoring it. A store that compresses or++ * encrypts its blobs (Stalwart) cannot pass that check: `verify: false` still archives and restores++ * the directory, but the drill checks only the archive.*/function normaliseObjects(raw, where) {if (raw == null) return [];@@@ -158,8 +164,38 @@ function normaliseObjects(raw, where) {if (!o || typeof o !== 'object' || Array.isArray(o)) throw new InventoryError(`${at} must be an object`);if (!FILE_NAME_RE.test(o.name || '')) throw new InventoryError(`${at}.name must be a plain name`);if (seen.has(o.name)) throw new InventoryError(`${at}.name "${o.name}" is listed twice`);++ if (o.verify != null && typeof o.verify !== 'boolean') throw new InventoryError(`${at}.verify must be true or false`);seen.add(o.name);-- return { name: o.name, path: abs(o.path, `${at}.path`) };++ return { name: o.name, path: abs(o.path, `${at}.path`), verify: o.verify !== false };++ });++}++++/**++ * `auxiliaryUnits`: service units the service depends on but does not run in its own process, and++ * that ovhost must not restart on every deploy. Each is installed from `unitSources`; `restartOn`++ * lists checkout paths (config, unit files) whose change is the only reason to restart it. The++ * deploy restarts an auxiliary unit before the service's own units, and a rollback never does.++ */++function normaliseAuxiliaryUnits(raw, where) {++ if (raw == null) return [];++ if (!Array.isArray(raw)) throw new InventoryError(`${where}.auxiliaryUnits must be a list of units`);++ const seen = new Set();++ return raw.map((a, i) => {++ const at = `${where}.auxiliaryUnits[${i}]`;++ const unit = typeof a === 'string' ? a : a && a.unit;++ if (typeof unit !== 'string' || !AUX_UNIT_RE.test(unit)) throw new InventoryError(`${at}.unit must be a .service unit`);++ if (seen.has(unit)) throw new InventoryError(`${at}.unit "${unit}" is listed twice`);++ seen.add(unit);++ const restartOn = typeof a === 'string' ? [] : (a.restartOn || []);++ if (!Array.isArray(restartOn)) throw new InventoryError(`${at}.restartOn must be a list of paths inside the checkout`);++ for (const f of restartOn) rel(f, `${at}.restartOn`);++ // applyOn: paths whose change a restart does not take (Stalwart reads its rendered configuration, so a change++ // to the template must be rendered and applied first). ovhost restarts nothing for them and says so.++ const applyOn = typeof a === 'string' ? [] : (a.applyOn || []);++ if (!Array.isArray(applyOn)) throw new InventoryError(`${at}.applyOn must be a list of paths inside the checkout`);++ for (const f of applyOn) rel(f, `${at}.applyOn`);++ const applyHint = typeof a === 'string' || a.applyHint == null ? null : String(a.applyHint).slice(0, 300);++ return { unit, restartOn, applyOn, applyHint };});}@@@ -212,6 +248,7 @@ function normaliseService(id, raw, defaults) {// OpenRestream's transport workers ([email protected], one instance per release), which// drain on their own. status/validate list their instances read-only.workerUnits: s.workerUnits || [],++ auxiliaryUnits: normaliseAuxiliaryUnits(s.auxiliaryUnits, where),socketUnit: s.socketUnit || null,unitSources: s.unitSources || {},envFile: s.envFile ? abs(s.envFile, `${where}.envFile`) : null,@@@ -283,8 +320,13 @@ function normaliseService(id, raw, defaults) {}}}++ for (const a of out.auxiliaryUnits) {++ if (out.units.includes(a.unit)) throw new InventoryError(`${where}.auxiliaryUnits: "${a.unit}" is already in units; an auxiliary unit is not restarted with the service`);++ if (out.workerUnits.includes(a.unit)) throw new InventoryError(`${where}.auxiliaryUnits: "${a.unit}" is a worker unit; declare it once`);++ if (!Object.hasOwn(out.unitSources, a.unit)) throw new InventoryError(`${where}.auxiliaryUnits.${a.unit}: needs a unitSources entry so the deploy can install and track it`);++ }for (const [unit, src] of Object.entries(out.unitSources)) {-- if (!UNIT_RE.test(unit) && !/^[A-Za-z0-9@._-]+\.(service|socket)\.d\/[A-Za-z0-9._-]+\.conf$/.test(unit)) throw new InventoryError(`${where}.unitSources: "${unit}" is not a unit or drop-in name`);++ if (!SOURCE_UNIT_RE.test(unit) && !/^[A-Za-z0-9@._-]+\.(service|socket)\.d\/[A-Za-z0-9._-]+\.conf$/.test(unit)) throw new InventoryError(`${where}.unitSources: "${unit}" is not a unit, slice or drop-in name`);rel(src, `${where}.unitSources.${unit}`);}if (out.port !== null && !(Number.isInteger(out.port) && out.port > 0 && out.port < 65536)) throw new InventoryError(`${where}.port must be a TCP port`);@@@ -256,6 +256,9 @@ async function computePlan(ctx, svc, { mode = 'deploy', to = null, fetch = true,preflight: preflight.describe(svc),restartNeeded,units: svc.units,++ auxiliaryUnits: svc.auxiliaryUnits.map((a) => a.unit),++ auxRestart: [],++ auxRestartNeeded: false,extraUnits: [],socketUnit: svc.socketUnit,socket,@@@ -268,6 +271,9 @@ async function computePlan(ctx, svc, { mode = 'deploy', to = null, fetch = true,installVhosts: false,vhostsRemoved: [],nginxNotInstalled: !!(svc.nginx && svc.nginx.repoVhost && changed.includes(svc.nginx.repoVhost)),++ nginxVhost: null,++ envFile: svc.envFile,++ databases: svc.databases.map((d) => d.database),keep: rel.keep,};}@@@ -194,6 +194,17 @@ async function computePlan(ctx, svc, { mode = 'deploy', to = null, fetch = true,}const units = [...svc.units, ...extraUnits];const restartNeeded = units.length > 0 && (restart || codeChanged.length > 0);++ // An auxiliary unit (Mail's Stalwart) is installed and listed with the service but is not restarted++ // on every deploy: only a change to one of its declared paths (its rendered config, its unit or++ // slice) is a reason. Rollbacks never restart it; the rendered config on disk did not change.++ const auxRestart = mode === 'deploy'++ ? svc.auxiliaryUnits.filter((a) => a.restartOn.length && changed.some((f) => matchesPath(f, a.restartOn))).map((a) => a.unit)++ : [];++ // A change a restart would not take (a configuration template that must be rendered and applied): restart++ // nothing, and say it is not live yet.++ const auxApply = mode === 'deploy'++ ? svc.auxiliaryUnits.filter((a) => (a.applyOn || []).length && changed.some((f) => matchesPath(f, a.applyOn))).map((a) => ({ unit: a.unit, hint: a.applyHint }))++ : [];const unitDrift = [];for (const [unit, src] of Object.entries(svc.unitSources)) {@@@ -240,6 +251,10 @@ async function computePlan(ctx, svc, { mode = 'deploy', to = null, fetch = true,preflight: preflight.describe(svc),restartNeeded,units,++ auxiliaryUnits: svc.auxiliaryUnits.map((a) => a.unit),++ auxRestart,++ auxRestartNeeded: auxRestart.length > 0,++ auxApply,extraUnits,skippedJobs,socketUnit: svc.socketUnit,@@@ -250,9 +265,14 @@ async function computePlan(ctx, svc, { mode = 'deploy', to = null, fetch = true,drainPolicy: svc.drain.policy,managed: svc.managed,installVhosts: !!(svc.nginx && svc.nginx.installOnDeploy),++ nginxVhost: svc.nginx && svc.nginx.installOnDeploy && svc.nginx.repoVhost++ ? { name: svc.nginx.vhost || path.basename(svc.nginx.repoVhost), source: svc.nginx.repoVhost }++ : null,vhostsRemoved,nginxNotInstalled: !!(svc.nginx && svc.nginx.repoVhost && !svc.nginx.installOnDeploy && changed.includes(svc.nginx.repoVhost)),announceFrom: svc.announce.releaseFiles,++ envFile: svc.envFile,++ databases: svc.databases.map((d) => d.database),};}@@@ -362,7 +382,9 @@ async function installUnits(ctx, svc, record) {}record.unitsInstalled = changed;if (svc.enableUnits) {-- for (const unit of svc.units) {++ // Auxiliary units are enabled but never started here: they are already running, or the++ // operator starts them once (Mail's Stalwart needs its rendered config first).++ for (const unit of [...svc.units, ...svc.auxiliaryUnits.map((a) => a.unit)]) {const enabled = await systemd.systemctl(ctx.exec, 'enable', unit);if (enabled.code !== 0) throw new Error(`systemctl enable ${unit} failed: ${enabled.stderr.trim()}`);}@@@ -546,6 +568,15 @@ async function apply(ctx, svc, mode, opts = {}) {if (opts.installUnits || svc.installUnits) await installUnits(ctx, svc, record);if (plan.backupNeeded && plan.restartNeeded) await backupBeforeRestart(ctx, svc, record);if (plan.restartNeeded) await guardSessions(ctx, svc, guardedOpts, record, 'immediately before the restart');++ for (const a of plan.auxApply || []) {++ log(`NOTE: ${a.unit}'s configuration changed and is NOT live until it is applied by hand${a.hint ? `: ${a.hint}` : ''}. ovhost restarts nothing for it.`);++ (record.auxApplyPending = record.auxApplyPending || []).push(a.unit);++ }++ for (const u of plan.auxRestart || []) {++ log(`restarting auxiliary ${u} (its configuration changed; ${svc.units.join(', ')} is not restarted for this)`);++ await systemd.restart(ctx.exec, u);++ (record.auxRestarted = record.auxRestarted || []).push(u);++ }} catch (err) {// Nothing has been restarted: put the files back the way the running process expects.if (record.checkoutMoved) {@@@ -560,7 +591,8 @@ async function apply(ctx, svc, mode, opts = {}) {const announcements = await releaseAnnouncements(ctx, svc);if (!plan.restartNeeded) {record.result = mode === 'deploy' ? 'deployed' : 'rolled-back';-- log(`${plan.units.length ? `only ${svc.noRestartPaths.join(' ')} changed — no restart needed` : 'no units to restart'}`);++ if (plan.auxRestartNeeded) log(`auxiliary ${plan.auxRestart.join(', ')} restarted; ${svc.units.join(', ')} not restarted`);++ else log(`${plan.units.length ? `only ${svc.noRestartPaths.join(' ')} changed — no restart needed` : 'no units to restart'}`);if (svc.ready) {const p = await readiness.probe(exec, svc);record.ready = p.ok;@@@ -20,7 +20,7 @@ runTests([// Every service on the host (2026-10-15): Sites retired, and the fourteen added since the first copy of this file.assert.deepStrictEqual(Object.keys(inv.services), ['network', 'live', 'media', 'tools', 'community', 'events', 'games', 'host', 'billing', 'chat', 'search', 'sources', 'wiki', 'blog', 'tips', 'vip', 'news', 'reviews', 'deals', 'coupons', 'trade', 'codes', 'ai', 'bot', 'openre','actor', 'food', 'help', 'work', 'quest', 'rent', 'media-hub', 'inventory', 'play', 'zone', 'watch', 'run', 'services', 'space',-- 'design', 'exchange', 'report', 'vision', 'fun', 'shoes', 'bench', 'cam', 'domains', 'meme', 'surf', 'today', 'tours']);++ 'design', 'exchange', 'report', 'vision', 'fun', 'shoes', 'bench', 'cam', 'domains', 'meme', 'surf', 'today', 'tours', 'mail']);assert.ok(Object.values(inv.services).filter((s) => s.databases.length).every((s) => s.drill), 'every service with a database has a restore drill');assert.strictEqual(inv.services.live.socketUnit, 'openvibe-live.socket');assert.ok(!inv.services.live.units.includes('openvibe-live.socket'));@@@ -42,8 +42,22 @@ runTests([assert.deepStrictEqual([inv.services.bot.nginx.vhost, inv.services.bot.nginx.repoVhost, inv.services.bot.nginx.installOnDeploy], ['openvibe.bot.conf', 'deploy/nginx/openvibe.bot.conf', true]);assert.strictEqual(inv.services.bot.installUnits, true);assert.strictEqual(inv.services.bot.enableUnits, true);++ // Mail: the Node API on loopback, Stalwart as an auxiliary unit that a deploy does not restart,++ // the database (Stalwart's data store too) and the message blobs outside the database.++ assert.deepStrictEqual([inv.services.mail.port, inv.services.mail.envFile, inv.services.mail.ready.url], [5250, '/etc/openvibe/mail.env', 'http://127.0.0.1:5250/api/ready']);++ assert.deepStrictEqual(inv.services.mail.units, ['openvibe-mail.service']);++ assert.deepStrictEqual(inv.services.mail.auxiliaryUnits.map(({ unit, restartOn, applyOn }) => ({ unit, restartOn, applyOn })),++ [{ unit: 'openvibe-stalwart.service', restartOn: ['deploy/systemd/openvibe-stalwart.service', 'deploy/systemd/openvibe-stalwart.slice'], applyOn: ['deploy/stalwart/'] }]);++ assert.match(inv.services.mail.auxiliaryUnits[0].applyHint, /render and apply/);++ assert.strictEqual(inv.services.mail.unitSources['openvibe-stalwart.slice'], 'deploy/systemd/openvibe-stalwart.slice');++ assert.deepStrictEqual([inv.services.mail.databases[0].database, inv.services.mail.databases[0].role], ['ov_mail', 'ov_mail']);++ assert.deepStrictEqual(inv.services.mail.objects, [{ name: 'stalwart-blobs', path: '/var/lib/openvibe-stalwart/blobs', verify: false }]);++ assert.deepStrictEqual([inv.services.mail.nginx.vhost, inv.services.mail.nginx.repoVhost, inv.services.mail.nginx.installOnDeploy], ['openvibe.email.conf', 'deploy/nginx/openvibe.email.conf', true]);++ assert.strictEqual(inv.services.mail.drill.supported, false);++ assert.ok(inv.services.mail.lifecycle.shutdown.deadlineSeconds <= 20, 'within openvibe-mail.service TimeoutStopSec');const text = JSON.stringify(raw);-- assert.ok(!/(sk_|ghp_|BEGIN [A-Z ]*PRIVATE KEY|password\s*[:=]\s*\S)/i.test(text), 'no secret values in the example');++ // Token-shaped values only: MAIL_MASK_DOMAIN is a name, not a secret (the old /i "sk_" caught it).++ assert.ok(!/(\bsk_[A-Za-z0-9]{8,}|\bghp_[A-Za-z0-9]{8,}|BEGIN [A-Z ]*PRIVATE KEY|password\s*[:=]\s*\S)/.test(text), 'no secret values in the example');}),test('the released service manifest is valid against the contracts schema', () => {nnew file mode 100644@@@ -0,0 +1,155 @@++'use strict';++/**++ * OpenVibe.Mail in the host inventory: the Node API on 5250, Stalwart as an auxiliary unit that a++ * normal Mail deploy never restarts, and the message blobs archived with the database.++ */++const assert = require('assert');++const fs = require('fs');++const path = require('path');++const { createFakeHost } = require('./fake-host');++const { main } = require('../lib/cli');++const { normalise } = require('../lib/inventory');++++const STATE = '/var/lib/openvibe-host';++++const MAIL_FILES = {++ 'package.json': JSON.stringify({ name: 'openvibe-mail', version: '0.1.0', dependencies: { express: '^4.21.2' } }, null, 2),++ 'package-lock.json': '{"lockfileVersion":3,"packages":{}}',++ 'server/index.js': 'console.log(1);',++ '.env.example': 'DATABASE_URL=\nDATABASE_DIRECT_URL=\n',++ 'deploy/systemd/openvibe-mail.service': '[Service]\nExecStart=/usr/bin/node server/index.js\nTimeoutStopSec=20\n',++ 'deploy/systemd/openvibe-stalwart.service': '[Service]\nExecStart=/opt/openvibe.stalwart/stalwart --config /etc/openvibe/stalwart/config.json\n',++ 'deploy/systemd/openvibe-stalwart.slice': '[Slice]\nMemoryMax=1G\n',++ 'deploy/stalwart/config.toml': '# Stalwart configuration\n',++ 'deploy/nginx/openvibe.email.conf': 'server { listen 443; }\n',++};++++function mailHost() {++ const host = createFakeHost();++ const example = JSON.parse(fs.readFileSync(path.join(__dirname, '..', 'host.example.json'), 'utf8'));++ const doc = {++ host: 'fake-host',++ stateDir: STATE,++ backupDir: '/var/backups/openvibe',++ defaults: { owner: 'ubuntu', branch: 'main' },++ services: { mail: example.services.mail },++ };++ host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });++ host.inv = normalise(doc);++ const svc = host.inv.services.mail;++ for (const db of svc.databases) host.pgDatabases.add(db.database);++ host.pgBackups = [{ type: 'diff', timestamp: { stop: Math.floor(host.exec.now() / 1000) - 3600 } }];++ host.ensureDir('/etc/nginx/sites-available');++ host.ensureDir('/etc/nginx/sites-enabled');++ host.badShas = new Set();++ host.cli = async (...argv) => {++ const lines = [];++ const code = await main(argv, { exec: host.exec, out: (s) => lines.push(s), env: {} });++ return { code, out: lines.join('\n') };++ };++++ const repo = host.createRepo(svc.repo, { owner: 'ubuntu' });++ const sha = repo.commit(MAIL_FILES, { message: 'initial' });++ repo.publish(sha);++ repo.checkout(sha);++ for (const dep of Object.keys(JSON.parse(MAIL_FILES['package.json']).dependencies)) {++ host.put(path.join(svc.repo, 'node_modules', dep, 'package.json'), JSON.stringify({ name: dep }), { owner: 'ubuntu' });++ }++ for (const u of [...svc.units, ...svc.auxiliaryUnits.map((a) => a.unit)]) host.addUnit(u);++ for (const [unit, src] of Object.entries(svc.unitSources)) host.put(path.join('/etc/systemd/system', unit), MAIL_FILES[src]);++ host.put(svc.envFile, svc.env.required.map((n) => `${n}=test`).join('\n'), { mode: 0o600 });++ host.http.set(svc.ready.url, () => ({ status: 200, body: { status: 'ok' } }));++ host.push = (changes, message = 'change') => { const s = repo.commit(changes, { message }); repo.publish(s); return s; };++ return host;++}++++function lastRecord(host, id) {++ const lines = host.read(`${STATE}/backups/${id}.jsonl`).trim().split('\n');++ return JSON.parse(lines[lines.length - 1]);++}++++(async () => {++ let failed = 0;++ const atest = async (name, fn) => {++ try { await fn(); console.log(` ✓ ${name}`); } catch (err) { failed += 1; console.log(` ✗ ${name}\n${err.stack || err}`); }++ };++++ await atest('plan names the Mail unit, Stalwart auxiliary unit, env, database and vhost', async () => {++ const host = mailHost();++ host.push({ 'server/index.js': 'console.log(2);' }, 'server change');++ const r = await host.cli('plan', 'mail');++ assert.strictEqual(r.code, 0, r.out);++ assert.match(r.out, /^ repo\s+\/opt\/openvibe\.mail$/m);++ assert.match(r.out, /^ env\s+\/etc\/openvibe\/mail\.env$/m);++ assert.match(r.out, /^ databases\s+ov_mail$/m);++ assert.match(r.out, /^ restart\s+yes: openvibe-mail\.service$/m);++ assert.match(r.out, /^ auxiliary\s+openvibe-stalwart\.service \(installed with the service; not restarted unless its config changes\)$/m);++ assert.match(r.out, /^ nginx\s+openvibe\.email\.conf from deploy\/nginx\/openvibe\.email\.conf installed \(nginx -t, reload\) on deploy$/m);++ assert.doesNotMatch(r.out, /^ restart\s+yes:.*openvibe-stalwart/m);++ });++++ await atest('a Mail code deploy restarts openvibe-mail.service and never Stalwart', async () => {++ const host = mailHost();++ host.push({ 'server/index.js': 'console.log(2);' }, 'server change');++ const r = await host.cli('deploy', 'mail');++ assert.strictEqual(r.code, 0, r.out);++ assert.deepStrictEqual(host.restarts(), ['openvibe-mail.service']);++ assert.strictEqual(host.units.get('openvibe-stalwart.service').restarts, 0);++ assert.ok(host.read('/etc/nginx/sites-available/openvibe.email.conf'), 'the repo vhost was installed');++ assert.ok(host.files.has('/etc/nginx/sites-enabled/openvibe.email.conf'), 'the vhost was linked into sites-enabled');++ });++++ await atest('a Stalwart template change restarts nothing and says it is not live until applied', async () => {++ const host = mailHost();++ host.push({ 'deploy/stalwart/config.toml': '# Stalwart configuration, changed\n' }, 'mail server change');++ const r = await host.cli('deploy', 'mail');++ assert.strictEqual(r.code, 0, r.out);++ assert.deepStrictEqual(host.restarts(), [], 'a restart would keep the old configuration, so nothing restarts');++ assert.match(r.out, /openvibe-stalwart\.service's configuration changed and is NOT live until it is applied by hand: render and apply/);++ const [rec] = (host.read(`${STATE}/releases/mail.jsonl`) || '').trim().split('\n').map((l) => JSON.parse(l));++ assert.deepStrictEqual(rec.auxApplyPending, ['openvibe-stalwart.service']);++ assert.strictEqual(rec.restarted, false);++ });++++ await atest('a Stalwart unit change restarts only the auxiliary unit', async () => {++ const host = mailHost();++ host.push({ 'deploy/systemd/openvibe-stalwart.slice': '[Slice]\nMemoryMax=1G\nCPUQuota=100%\n# changed\n' }, 'slice change');++ const r = await host.cli('deploy', 'mail');++ assert.strictEqual(r.code, 0, r.out);++ assert.deepStrictEqual(host.restarts(), ['openvibe-stalwart.service']);++ assert.strictEqual(host.units.get('openvibe-mail.service').restarts, 0);++ assert.match(r.out, /restarting auxiliary openvibe-stalwart\.service/);++ const [rec] = (host.read(`${STATE}/releases/mail.jsonl`) || '').trim().split('\n').map((l) => JSON.parse(l));++ assert.deepStrictEqual(rec.auxRestarted, ['openvibe-stalwart.service']);++ });++++ await atest('a docs-only change restarts nothing', async () => {++ const host = mailHost();++ host.push({ 'docs/INSTALL.md': 'changed\n' }, 'docs change');++ const r = await host.cli('deploy', 'mail');++ assert.strictEqual(r.code, 0, r.out);++ assert.deepStrictEqual(host.restarts(), []);++ });++++ await atest('backup archives ov_mail and the Stalwart blob directory together', async () => {++ const host = mailHost();++ host.put('/var/lib/openvibe-stalwart/blobs/aa/bb/0123456789abcdef', 'compressed message bytes', { owner: 'openvibe', mode: 0o600 });++ const r = await host.cli('backup', 'mail', '--logical');++ assert.strictEqual(r.code, 0, r.out);++ const rec = lastRecord(host, 'mail');++ assert.strictEqual(rec.ok, true);++ assert.strictEqual(host.dumps[0].database, 'ov_mail');++ const names = rec.files.map((f) => f.file).sort();++ assert.deepStrictEqual(names, ['mail.dump', 'stalwart-blobs.tar.gz']);++ assert.ok(host.files.has(`${rec.dir}/stalwart-blobs.tar.gz`));++ });++++ await atest('a non-content-addressed object store must say verify:false', () => {++ const host = mailHost();++ assert.strictEqual(normalise({ services: { mail: host.inv.services.mail } }).services.mail.objects[0].verify, false);++ assert.throws(() => normalise({ services: { x: { repo: '/opt/x', owner: 'ubuntu', objects: [{ name: 'b', path: '/var/lib/b', verify: 'yes' }] } } }), /verify must be true or false/);++ });++++ if (failed) { console.error(`${failed} failed`); process.exit(1); }++ console.log('all mail inventory tests passed');++})().catch((err) => { console.error(err); process.exit(1); });
@@ -288,7 +288,7 @@ The sandbox blocks writes outside the drill directory and addresses beyond loopb
@@ -288,7 +288,7 @@ The sandbox blocks writes outside the drill directory and addresses beyond loopb
### Inventory
### Inventory
-`host.example.json` describes every service on the host. It records the checkout, owner/`runAs`, units and socket unit, `unitSources` (repo unit files), env file, `env.required` (names, or `"from-example"`), port, loopback `ready` URL (plus headers, since Tools needs `Host: openvibe.tools`), `packages` (`"apps/*"` for Tools), install command, `build` hooks, `noRestartPaths`, the `protected` probe (`http-json-count` — a numeric `field`, plus any present `addFields` —, `postgresql-count`, or `sum` of several: Live streams, Media recordings, chunk sessions and uploads, Events SSE connections, Chat sockets, Games players online, OpenRestream ingest sessions, Tools running jobs), the `drain` policy, `databases` (each a PostgreSQL database `{ "name", "engine": "postgresql", "database": "ov_<name>" }`; [docs/db-inventory.md](docs/db-inventory.md)), `backupOnChange`, `nginx`, `layout` (`release` for a `releases/<id>` + `current` service using the `release-layout` strategy) and `workerUnits`. `workerUnits` are units, or templates such as OpenRestream's `[email protected]`, that belong to the service but that ovhost **never starts, stops or restarts**: they drain and exit on their own. `status` and `validate` list their instances read-only, `validate` warns when none runs, and the inventory refuses a worker unit, or an instance of one, in `units`. An `addFields` entry that the service does not report at all is read as zero (an older service), while one it reports as `null` is read as unknown: the service could not answer it, so the restart is held. The real file is `/etc/openvibe/host.json` and is **never committed**. When `ovhost` runs as root it refuses an inventory that is not root-owned or that anyone else can write, and it rejects relative paths, non-loopback probe URLs, non-SELECT probe queries and vhost names that contain a path.
+`host.example.json` describes every service on the host. It records the checkout, owner/`runAs`, units and socket unit, `unitSources` (repo unit files), env file, `env.required` (names, or `"from-example"`), port, loopback `ready` URL (plus headers, since Tools needs `Host: openvibe.tools`), `packages` (`"apps/*"` for Tools), install command, `build` hooks, `noRestartPaths`, the `protected` probe (`http-json-count` — a numeric `field`, plus any present `addFields` —, `postgresql-count`, or `sum` of several: Live streams, Media recordings, chunk sessions and uploads, Events SSE connections, Chat sockets, Games players online, OpenRestream ingest sessions, Tools running jobs), the `drain` policy, `databases` (each a PostgreSQL database `{ "name", "engine": "postgresql", "database": "ov_<name>" }`; [docs/db-inventory.md](docs/db-inventory.md)), `backupOnChange`, `nginx`, `layout` (`release` for a `releases/<id>` + `current` service using the `release-layout` strategy) and `workerUnits`. `workerUnits` are units, or templates such as OpenRestream's `[email protected]`, that belong to the service but that ovhost **never starts, stops or restarts**: they drain and exit on their own. `status` and `validate` list their instances read-only, `validate` warns when none runs, and the inventory refuses a worker unit, or an instance of one, in `units`. `auxiliaryUnits` are service units a service depends on but does not run in its own process: Mail's `openvibe-stalwart.service`, installed from `unitSources` (which may also carry its `.slice`). A deploy restarts one only when a path in its `restartOn` list changed, and a rollback never does. `objects` directories are archived by `backup`; when a store compresses or encrypts its blobs, declare `"verify": false` so a drill checks the archive but not each file's sha256. An `addFields` entry that the service does not report at all is read as zero (an older service), while one it reports as `null` is read as unknown: the service could not answer it, so the restart is held. The real file is `/etc/openvibe/host.json` and is **never committed**. When `ovhost` runs as root it refuses an inventory that is not root-owned or that anyone else can write, and it rejects relative paths, non-loopback probe URLs, non-SELECT probe queries and vhost names that contain a path.
**Lifecycle** (roadmap WS-P task 1, `lib/lifecycle.js`). Each service's lifecycle is the `lifecycle` block of its manifest in openvibe-contracts (≥ 0.55.0): liveness, shutdown (signal, `deadlineSeconds`, drains, workers), startupRecovery, rollback, contracts and leases. A service entry may carry its own `lifecycle` block, which replaces the manifest's on this host, and `validate --manifest <file>` reads another manifest file. `ovhost validate` fails when no block is found or a field is missing, naming the service and the field (`live: lifecycle.shutdown.deadlineSeconds is missing`). It also fails when `deadlineSeconds` exceeds a unit's stop timeout (systemd's effective `TimeoutStopUSec`, else `TimeoutStopSec` in the unit source) or the unit's `KillSignal` is another signal, and when the checkout's installed openvibe-contracts is outside `contracts.range`. Worker units that drain longer than their stop timeout are a warning, since ovhost never stops them. This repository still pins openvibe-contracts v0.49.0, which has no lifecycle blocks, so until that pin moves to v0.55.0 or later `validate` reports every service as undeclared.
**Lifecycle** (roadmap WS-P task 1, `lib/lifecycle.js`). Each service's lifecycle is the `lifecycle` block of its manifest in openvibe-contracts (≥ 0.55.0): liveness, shutdown (signal, `deadlineSeconds`, drains, workers), startupRecovery, rollback, contracts and leases. A service entry may carry its own `lifecycle` block, which replaces the manifest's on this host, and `validate --manifest <file>` reads another manifest file. `ovhost validate` fails when no block is found or a field is missing, naming the service and the field (`live: lifecycle.shutdown.deadlineSeconds is missing`). It also fails when `deadlineSeconds` exceeds a unit's stop timeout (systemd's effective `TimeoutStopUSec`, else `TimeoutStopSec` in the unit source) or the unit's `KillSignal` is another signal, and when the checkout's installed openvibe-contracts is outside `contracts.range`. Worker units that drain longer than their stop timeout are a warning, since ovhost never stops them. This repository still pins openvibe-contracts v0.49.0, which has no lifecycle blocks, so until that pin moves to v0.55.0 or later `validate` reports every service as undeclared.
@@ -11,7 +11,9 @@ Restore drills (a second instance started from a backup and compared with produc
@@ -11,7 +11,9 @@ Restore drills (a second instance started from a backup and compared with produc
## What is backed up
## What is backed up
-Every inventory database is PostgreSQL. `ovhost backup --all` verifies the cluster's pgBackRest backups and WAL archiver on every run. It makes a per-service `pg_dump -Fc` when the last good logical dump is over seven days old or when `--logical` is given. Declared content-addressed object directories are archived as `.tar.gz` files. Restore drills use the logical dumps and object archives.
+Every inventory database is PostgreSQL. `ovhost backup --all` verifies the cluster's pgBackRest backups and WAL archiver on every run. It makes a per-service `pg_dump -Fc` when the last good logical dump is over seven days old or when `--logical` is given. Declared object directories are archived as `.tar.gz` files. Restore drills use the logical dumps and object archives.
+
+A directory whose files are named by their own sha256 is content-addressed and a drill verifies every blob. Some stores compress or encrypt their blobs (Stalwart's message blobs under `/var/lib/openvibe-stalwart/blobs`), so the bytes do not hash to the file name: declare `"verify": false` in the inventory's `objects` entry. The directory is still archived, uploaded and restored; the drill checks the archive, not each blob.
## One run
## One run
@@ -13,3 +13,8 @@ Every `services.<id>.databases[]` entry names a database in the host PostgreSQL
@@ -13,3 +13,8 @@ Every `services.<id>.databases[]` entry names a database in the host PostgreSQL
```
```
The drill creates a scratch database and role, restores the archive, and points both settings at the scratch database. A `postgresql-count` protected-session probe runs a read-only SELECT through the postgres OS user.
The drill creates a scratch database and role, restores the archive, and points both settings at the scratch database. A `postgresql-count` protected-session probe runs a read-only SELECT through the postgres OS user.
+
+Mail's `ov_mail` is both the Mail service's database and Stalwart's data store; Stalwart's tables
+live in the same `public` schema. `sudo ovhost data provision mail` creates the `ov_mail` owner and
+`ov_mail_app` runtime roles. Stalwart connects directly as the owner (`STALWART_DB_USER=ov_mail`,
+`STALWART_DB_PASSWORD`); Mail's migrations run on `DATABASE_DIRECT_URL`, the same owner on :5432.
@@ -47,6 +47,8 @@ Repository wrappers call `ovhost plan`, `ovhost deploy`, or `ovhost rollback` fo
@@ -47,6 +47,8 @@ Repository wrappers call `ovhost plan`, `ovhost deploy`, or `ovhost rollback` fo
nothing restarted (exit 2).
nothing restarted (exit 2).
7. Restart the units (plus unit files matching `unitsMatch`), wait for readiness (`ready.release`,
7. Restart the units (plus unit files matching `unitsMatch`), wait for readiness (`ready.release`,
`ready.allUnits`), and on failure roll back: restore, reinstall, rebuild, restart (exit 3, or 4).
`ready.allUnits`), and on failure roll back: restore, reinstall, rebuild, restart (exit 3, or 4).
+ An auxiliary unit (`auxiliaryUnits`, Mail's Stalwart) is restarted first, and only when one of its
+ `restartOn` paths changed; the service itself is not restarted for a Stalwart-only change.
8. Record the release; announce it (Sites: one notification per `dist/<domain>/release.json`).
8. Record the release; announce it (Sites: one notification per `dist/<domain>/release.json`).
### Release layout (release-layout)
### Release layout (release-layout)
@@ -105,6 +107,7 @@ overrides them.
@@ -105,6 +107,7 @@ overrides them.
| `generated` | tracked paths a build rewrites; restored from git before the merge, never a local change | static-build: `["dist/"]` |
| `generated` | tracked paths a build rewrites; restored from git before the merge, never a local change | static-build: `["dist/"]` |
| `installUnits` | install drifted `unitSources` on every deploy (as `--install-units`) | false |
| `installUnits` | install drifted `unitSources` on every deploy (as `--install-units`) | false |
| `unitsMatch` | also restart unit files matching this glob (named when the inventory lacks them) | none |
| `unitsMatch` | also restart unit files matching this glob (named when the inventory lacks them) | none |
+| `auxiliaryUnits` | `[{ unit, restartOn: [paths] }]` units the service depends on but that an ordinary deploy must not restart (Mail's Stalwart). Installed from `unitSources`, listed by `status`/`validate`; restarted before the service only when a `restartOn` path changed. A rollback never restarts them | `[]` |
| `announce.releaseFiles` | one notification per `<dir>/*/release.json` instead of one for the service | static-build: `dist/*/release.json` |
| `announce.releaseFiles` | one notification per `<dir>/*/release.json` instead of one for the service | static-build: `dist/*/release.json` |
| `install.lockfile` | the lockfile that decides an install | `package-lock.json` (pnpm-build: `pnpm-lock.yaml`) |
| `install.lockfile` | the lockfile that decides an install | `package-lock.json` (pnpm-build: `pnpm-lock.yaml`) |
| `install.workspace` | one install at the root for a workspace | pnpm-build: true |
| `install.workspace` | one install at the root for a workspace | pnpm-build: true |
@@ -34,3 +34,18 @@ Why it is not switched on without the owner:
@@ -34,3 +34,18 @@ Why it is not switched on without the owner:
- the provider's edge firewall may already filter some of these ports.
- the provider's edge firewall may already filter some of these ports.
Enable it at a quiet time, with a second SSH session open, then run the browser check and a WebRTC publish and view.
Enable it at a quiet time, with a second SSH session open, then run the browser check and a WebRTC publish and view.
+
+## OpenVibe.Mail (M1, 2026-11-07)
+
+Inbound **TCP 25 only**. Stalwart M1 is receive-only: the rendered plan removes its submission
+(587), SMTPS (465), POP3, Sieve and HTTPS listeners before they open. JMAP/management (8080) and
+IMAP (143) stay on loopback; the Mail API (5250) is loopback behind nginx.
+
+When the default-deny proposal above is enabled, add only:
+
+```
+sudo ufw allow 25/tcp
+```
+
+Do **not** add 587 or 465. If `sudo ss -ltnp | grep -E ':(587|465|993|995|4190)\b'` names Stalwart,
+stop it and re-apply `deploy/stalwart/apply.js` before opening anything.
@@ -4003,6 +4003,146 @@
@@ -4003,6 +4003,146 @@
"countsTolerance": 5,
"countsTolerance": 5,
"_notes": "The restored copy boots with no service credentials (no tokens, no outbox publishing or subscriptions), Valkey on a closed port and every outbound service URL on a closed port; counts and readiness."
"_notes": "The restored copy boots with no service credentials (no tokens, no outbox publishing or subscriptions), Valkey on a closed port and every outbound service URL on a closed port; counts and readiness."
}
}
+ },
+ "mail": {
+ "_note": "OpenVibe.Mail (M1): receive-only Stalwart on inbound TCP 25, masks, burners and agent inboxes; database ov_mail; public door openvibe.email. openvibe-stalwart.service and its slice are auxiliary: installed from this checkout and listed by status/validate, but a Mail deploy restarts them only when deploy/stalwart/config.toml or one of their unit files changes. The binary is installed by hand from scripts/fetch-stalwart.js (pinned sha256) — ovhost never builds or downloads it. Message blobs at /var/lib/openvibe-stalwart/blobs are compressed and named by content hash, so they are archived by backup with verify:false and not sha256-checked. Remove openvibe.email from /etc/nginx/sites-enabled/parked-domains.conf before the vhost can win the name. Ports 587 and 465 stay closed.",
+ "repo": "/opt/openvibe.mail",
+ "units": [
+ "openvibe-mail.service"
+ ],
+ "auxiliaryUnits": [
+ {
+ "unit": "openvibe-stalwart.service",
+ "restartOn": [
+ "deploy/systemd/openvibe-stalwart.service",
+ "deploy/systemd/openvibe-stalwart.slice"
+ ],
+ "applyOn": [
+ "deploy/stalwart/"
+ ],
+ "applyHint": "render and apply it as OpenVibe.Mail docs/INSTALL.md says (render.js, then apply.js with the plan); a restart alone keeps the old configuration"
+ }
+ ],
+ "unitSources": {
+ "openvibe-mail.service": "deploy/systemd/openvibe-mail.service",
+ "openvibe-stalwart.service": "deploy/systemd/openvibe-stalwart.service",
+ "openvibe-stalwart.slice": "deploy/systemd/openvibe-stalwart.slice"
+ },
+ "installUnits": true,
+ "enableUnits": true,
+ "envFile": "/etc/openvibe/mail.env",
+ "env": {
+ "required": [
+ "NODE_ENV",
+ "PORT",
+ "HOST",
+ "BASE_URL",
+ "TRUST_PROXY",
+ "DATABASE_URL",
+ "DATABASE_DIRECT_URL",
+ "VALKEY_URL",
+ "VALKEY_PREFIX",
+ "OV_NETWORK_URL",
+ "OV_NETWORK_INTERNAL_URL",
+ "OV_OAUTH_CLIENT_ID",
+ "OV_OAUTH_CLIENT_SECRET",
+ "MAIL_BASE_DOMAIN",
+ "MAIL_MASK_DOMAIN",
+ "BURNER_DOMAINS",
+ "STALWART_URL",
+ "STALWART_ADMIN_USER",
+ "STALWART_ADMIN_PASSWORD"
+ ]
+ },
+ "port": 5250,
+ "health": {
+ "url": "http://127.0.0.1:5250/api/health"
+ },
+ "ready": {
+ "url": "http://127.0.0.1:5250/api/ready",
+ "timeoutSeconds": 30
+ },
+ "noRestartPaths": [
+ "docs/",
+ "test/",
+ "**/*.md",
+ "deploy/stalwart/",
+ "deploy/systemd/openvibe-stalwart.service",
+ "deploy/systemd/openvibe-stalwart.slice"
+ ],
+ "databases": [
+ {
+ "name": "mail",
+ "engine": "postgresql",
+ "database": "ov_mail",
+ "role": "ov_mail"
+ }
+ ],
+ "objects": [
+ {
+ "name": "stalwart-blobs",
+ "path": "/var/lib/openvibe-stalwart/blobs",
+ "verify": false
+ }
+ ],
+ "drill": {
+ "supported": false,
+ "reason": "M1 has no drill switch: readiness depends on the live Stalwart directory and the process writes the directory view at boot, so a second instance is not side-effect free."
+ },
+ "nginx": {
+ "vhost": "openvibe.email.conf",
+ "repoVhost": "deploy/nginx/openvibe.email.conf",
+ "installOnDeploy": true
+ },
+ "lifecycle": {
+ "liveness": {
+ "endpoint": "/api/health",
+ "means": "the process answers HTTP; nothing else is checked"
+ },
+ "shutdown": {
+ "signal": "SIGTERM",
+ "deadlineSeconds": 10,
+ "drains": [
+ "the HTTP server closes",
+ "the purge job stops",
+ "the database connection closes"
+ ]
+ },
+ "startupRecovery": {
+ "resumes": [
+ {
+ "kind": "jobs",
+ "what": "due mask expiries and burner purges run on the next interval"
+ },
+ {
+ "kind": "outbox",
+ "what": "unsent mail.received and mail.mask.created events stay in the outbox until the relay starts"
+ }
+ ]
+ },
+ "rollback": {
+ "conditions": [
+ "automatic: not ready within ready.timeoutSeconds"
+ ],
+ "window": "while ovhost waits for readiness",
+ "blockers": {
+ "none": "the service's migrations only add; Stalwart's tables live in the same database and are not touched by a Mail release"
+ }
+ },
+ "contracts": {
+ "range": ">=0.329.0 <1.0.0"
+ },
+ "leases": {
+ "claims": [
+ {
+ "what": "the purge job lease",
+ "holder": "the one mail process",
+ "expires": "MAIL_PURGE_INTERVAL_MS (60 s)",
+ "fencing": "a conditional UPDATE on mail_settings.updated_at"
+ }
+ ]
+ }
+ }
}
}
}
}
}
}
@@ -270,9 +270,9 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc
@@ -270,9 +270,9 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc
// The inventory as ovhost reads it (defaults applied). It holds no secret values.
// The inventory as ovhost reads it (defaults applied). It holds no secret values.
const ids = pos.slice(1);
const ids = pos.slice(1);
const list = ids.length ? ids.map((id) => lib.inventory.service(inv, id)) : Object.values(inv.services);
const list = ids.length ? ids.map((id) => lib.inventory.service(inv, id)) : Object.values(inv.services);
- const rows = list.map((s) => ({ id: s.id, managed: s.managed, repo: s.repo, owner: s.owner, runAs: s.runAs, branch: s.branch, units: s.units, socketUnit: s.socketUnit || null, workerUnits: s.workerUnits, port: s.port || null, ready: s.ready ? s.ready.url : null, protected: s.protected ? s.protected.kind : null, drain: s.drain ? s.drain.policy : null, layout: s.layout }));
+ const rows = list.map((s) => ({ id: s.id, managed: s.managed, repo: s.repo, owner: s.owner, runAs: s.runAs, branch: s.branch, units: s.units, socketUnit: s.socketUnit || null, workerUnits: s.workerUnits, auxiliaryUnits: s.auxiliaryUnits.map((a) => a.unit), port: s.port || null, ready: s.ready ? s.ready.url : null, protected: s.protected ? s.protected.kind : null, drain: s.drain ? s.drain.policy : null, layout: s.layout }));
if (json) { emit(ids.length === 1 ? rows[0] : rows); return 0; }
if (json) { emit(ids.length === 1 ? rows[0] : rows); return 0; }
- for (const r of rows) out(`${r.id.padEnd(10)} ${r.repo} units: ${r.units.join(' ') || '-'}${r.socketUnit ? ` socket: ${r.socketUnit}` : ''}${r.workerUnits.length ? ` workers: ${r.workerUnits.join(' ')}` : ''}${r.protected ? ` protected: ${r.protected}` : ''}${r.managed ? '' : ' [unmanaged]'}`);
+ for (const r of rows) out(`${r.id.padEnd(10)} ${r.repo} units: ${r.units.join(' ') || '-'}${r.socketUnit ? ` socket: ${r.socketUnit}` : ''}${r.workerUnits.length ? ` workers: ${r.workerUnits.join(' ')}` : ''}${r.auxiliaryUnits.length ? ` auxiliary: ${r.auxiliaryUnits.join(' ')}` : ''}${r.protected ? ` protected: ${r.protected}` : ''}${r.managed ? '' : ' [unmanaged]'}`);
return 0;
return 0;
}
}
case 'env-names': {
case 'env-names': {
@@ -301,6 +301,9 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc
@@ -301,6 +301,9 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc
if (json) { emit(p); return 0; }
if (json) { emit(p); return 0; }
out(`${p.service}: ${short(p.from)} → ${short(p.to)}${p.upToDate ? ' (up to date)' : ''} on ${p.branch}${p.branch !== p.expectedBranch ? ` (EXPECTED ${p.expectedBranch})` : ''}`);
out(`${p.service}: ${short(p.from)} → ${short(p.to)}${p.upToDate ? ' (up to date)' : ''} on ${p.branch}${p.branch !== p.expectedBranch ? ` (EXPECTED ${p.expectedBranch})` : ''}`);
out(` strategy ${p.strategy || 'none'}${p.layout === 'release' ? ` (release layout, keep ${p.keep})` : ''}`);
out(` strategy ${p.strategy || 'none'}${p.layout === 'release' ? ` (release layout, keep ${p.keep})` : ''}`);
+ out(` repo ${p.repo}`);
+ if (p.envFile) out(` env ${p.envFile}`);
+ if (p.databases && p.databases.length) out(` databases ${p.databases.join(', ')}`);
if (p.layout === 'release') out(` release ${p.fromRelease} → ${p.upToDate ? p.fromRelease : p.reuse ? `${p.toRelease} (exists; reused)` : `${p.toRelease} (new, prepared while ${p.fromRelease} serves)`}`);
if (p.layout === 'release') out(` release ${p.fromRelease} → ${p.upToDate ? p.fromRelease : p.reuse ? `${p.toRelease} (exists; reused)` : `${p.toRelease} (new, prepared while ${p.fromRelease} serves)`}`);
for (const c of p.commits.slice(0, 30)) out(` ${c}`);
for (const c of p.commits.slice(0, 30)) out(` ${c}`);
out(` files changed ${p.changedFiles} (${p.codeChanged} outside the no-restart paths)`);
out(` files changed ${p.changedFiles} (${p.codeChanged} outside the no-restart paths)`);
@@ -312,6 +315,7 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc
@@ -312,6 +315,7 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc
if (p.build && p.build.length) out(` build ${p.build.join(' && ')}`);
if (p.build && p.build.length) out(` build ${p.build.join(' && ')}`);
if (p.preflight && p.preflight.length) out(` preflight ${p.preflight.join('; ')}`);
if (p.preflight && p.preflight.length) out(` preflight ${p.preflight.join('; ')}`);
out(` restart ${p.restartNeeded ? `yes: ${p.units.join(', ')}${p.socketUnit ? ` (socket ${p.socketUnit} stays up)` : ''}` : p.layout === 'release' && !p.upToDate ? 'no — current switches, the process keeps running' : 'no'}`);
out(` restart ${p.restartNeeded ? `yes: ${p.units.join(', ')}${p.socketUnit ? ` (socket ${p.socketUnit} stays up)` : ''}` : p.layout === 'release' && !p.upToDate ? 'no — current switches, the process keeps running' : 'no'}`);
+ if (p.auxiliaryUnits && p.auxiliaryUnits.length) out(` auxiliary ${p.auxiliaryUnits.join(', ')} (installed with the service; not restarted unless its config changes${p.auxRestartNeeded ? ` — restarting now: ${p.auxRestart.join(', ')}` : ''})`);
if (p.extraUnits && p.extraUnits.length) out(` units ${p.extraUnits.join(', ')} are not in the inventory but are restarted too (unitsMatch)`);
if (p.extraUnits && p.extraUnits.length) out(` units ${p.extraUnits.join(', ')} are not in the inventory but are restarted too (unitsMatch)`);
if (p.skippedJobs && p.skippedJobs.length) out(` jobs ${p.skippedJobs.join(', ')} match unitsMatch but are oneshot jobs: not restarted, not waited on`);
if (p.skippedJobs && p.skippedJobs.length) out(` jobs ${p.skippedJobs.join(', ')} match unitsMatch but are oneshot jobs: not restarted, not waited on`);
if (p.socket) out(` socket ${p.socket.held ? `held by systemd on :${p.socket.port}` : p.socket.held === false ? `NOT held by systemd on :${p.socket.port} — a restart rebinds it (stop, restart the socket, start)` : 'unknown'}${p.socket.changed ? '; the socket unit file changed — a restart rebinds it' : ''}`);
if (p.socket) out(` socket ${p.socket.held ? `held by systemd on :${p.socket.port}` : p.socket.held === false ? `NOT held by systemd on :${p.socket.port} — a restart rebinds it (stop, restart the socket, start)` : 'unknown'}${p.socket.changed ? '; the socket unit file changed — a restart rebinds it' : ''}`);
@@ -321,7 +325,7 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc
@@ -321,7 +325,7 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc
}
}
if (p.backupNeeded) out(' backup schema files changed — databases are backed up before the restart');
if (p.backupNeeded) out(' backup schema files changed — databases are backed up before the restart');
if (p.unitDrift.length) out(` unit files differ from the repo: ${p.unitDrift.map((u) => u.unit).join(', ')}${p.installUnits ? ' (installed by the deploy)' : ' (--install-units)'}`);
if (p.unitDrift.length) out(` unit files differ from the repo: ${p.unitDrift.map((u) => u.unit).join(', ')}${p.installUnits ? ' (installed by the deploy)' : ' (--install-units)'}`);
- if (p.installVhosts) out(' nginx repo vhosts are installed (nginx -t, reload) on deploy');
+ if (p.installVhosts) out(` nginx ${p.nginxVhost ? `${p.nginxVhost.name} from ${p.nginxVhost.source} installed` : 'repo vhosts installed'} (nginx -t, reload) on deploy`);
for (const v of p.vhostsRemoved || []) out(` nginx ${v} left the repository; it stays installed (remove by hand)`);
for (const v of p.vhostsRemoved || []) out(` nginx ${v} left the repository; it stays installed (remove by hand)`);
if (p.nginxNotInstalled) out(' nginx the repo vhost changed and is NOT installed by a deploy (by hand: nginx -t, reload)');
if (p.nginxNotInstalled) out(' nginx the repo vhost changed and is NOT installed by a deploy (by hand: nginx -t, reload)');
if (p.announceFrom) out(` announce one release notification per ${p.announceFrom}`);
if (p.announceFrom) out(` announce one release notification per ${p.announceFrom}`);
@@ -386,9 +390,10 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc
@@ -386,9 +390,10 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc
const units = r.units.map((u) => `${u.unit.replace(/\.service$/, '')}=${u.active}${u.restarts ? `(${u.restarts} restarts)` : ''}`).join(' ');
const units = r.units.map((u) => `${u.unit.replace(/\.service$/, '')}=${u.active}${u.restarts ? `(${u.restarts} restarts)` : ''}`).join(' ');
const sock = r.socket ? ` socket=${r.socket.active}` : '';
const sock = r.socket ? ` socket=${r.socket.active}` : '';
const workers = r.workers && r.workers.length ? ` workers[${r.workers.map((w) => `${w.unit.replace(/\.service$/, '')}=${w.active}`).join(' ')}]` : '';
const workers = r.workers && r.workers.length ? ` workers[${r.workers.map((w) => `${w.unit.replace(/\.service$/, '')}=${w.active}`).join(' ')}]` : '';
+ const auxiliary = r.auxiliary && r.auxiliary.length ? ` aux[${r.auxiliary.map((a) => `${a.unit.replace(/\.service$/, '')}=${a.active}`).join(' ')}]` : '';
const ready = r.ready ? (r.ready.ok ? 'ready' : `NOT READY (${r.ready.status || r.ready.error})`) : '-';
const ready = r.ready ? (r.ready.ok ? 'ready' : `NOT READY (${r.ready.status || r.ready.error})`) : '-';
const prot = r.protected ? (r.protected.count == null ? ` ${r.protected.label}=unknown` : ` ${r.protected.label}=${r.protected.count}`) : '';
const prot = r.protected ? (r.protected.count == null ? ` ${r.protected.label}=unknown` : ` ${r.protected.label}=${r.protected.count}`) : '';
- out(`${r.service.padEnd(10)} ${short(r.sha)} ${ready.padEnd(12)} ${units || '(no units)'}${sock}${workers}${prot}${r.managed ? '' : ' [unmanaged]'}`);
+ out(`${r.service.padEnd(10)} ${short(r.sha)} ${ready.padEnd(12)} ${units || '(no units)'}${sock}${workers}${auxiliary}${prot}${r.managed ? '' : ' [unmanaged]'}`);
}
}
return 0;
return 0;
}
}
@@ -260,7 +260,7 @@ async function resolveBackup(exec, inv, svc, dirArg) {
@@ -260,7 +260,7 @@ async function resolveBackup(exec, inv, svc, dirArg) {
// to extract. A --backup directory has no record; its compressed size is the best we can do.
// to extract. A --backup directory has no record; its compressed size is the best we can do.
const rec = record && (record.files || []).find((f) => f.file === `${obj.name}.tar.gz`);
const rec = record && (record.files || []).find((f) => f.file === `${obj.name}.tar.gz`);
const sourceBytes = rec && Number.isFinite(rec.sourceBytes) ? rec.sourceBytes : fst.size;
const sourceBytes = rec && Number.isFinite(rec.sourceBytes) ? rec.sourceBytes : fst.size;
- objects.push({ name: obj.name, file, production: obj.path, sourceBytes });
+ objects.push({ name: obj.name, file, production: obj.path, sourceBytes, verify: obj.verify !== false });
}
}
return { dir, takenAt, files, objects };
return { dir, takenAt, files, objects };
}
}
@@ -468,9 +468,15 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
@@ -468,9 +468,15 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
const entry = { name: obj.name, source: obj.file, copy: dest, blobs: 0, skipped: 0, bytes: 0, mismatches: 0, integrity: null };
const entry = { name: obj.name, source: obj.file, copy: dest, blobs: 0, skipped: 0, bytes: 0, mismatches: 0, integrity: null };
record.objects.push(entry);
record.objects.push(entry);
log(`restoring object directory ${obj.file} → ${dest} (${svc.runAs})`);
log(`restoring object directory ${obj.file} → ${dest} (${svc.runAs})`);
- try { await verifyObjectBlobs(exec, dest, entry); } catch (err) { entry.integrity = `error: ${err.message}`; throw fail('objects', `${obj.name}: ${err.message}`); }
- entry.integrity = 'sha256 ok';
- log(`object directory ${obj.name}: ${entry.blobs} blob(s) sha256 ok${entry.skipped ? `, ${entry.skipped} non-blob file(s) skipped` : ''}`);
+ if (obj.verify === false) {
+ // The store compresses or encrypts each blob, so a file does not hash to its name.
+ entry.integrity = 'archive only (blobs are not content-addressed on disk)';
+ log(`object directory ${obj.name}: restored (integrity is by the archive; its blobs are not content-addressed)`);
+ } else {
+ try { await verifyObjectBlobs(exec, dest, entry); } catch (err) { entry.integrity = `error: ${err.message}`; throw fail('objects', `${obj.name}: ${err.message}`); }
+ entry.integrity = 'sha256 ok';
+ log(`object directory ${obj.name}: ${entry.blobs} blob(s) sha256 ok${entry.skipped ? `, ${entry.skipped} non-blob file(s) skipped` : ''}`);
+ }
}
}
const restoredCounts = [];
const restoredCounts = [];
for (const c of d.counts) {
for (const c of d.counts) {
@@ -671,7 +677,9 @@ function markdownRow(r) {
@@ -671,7 +677,9 @@ function markdownRow(r) {
for (const db of r.databases) checks.push(`\`pg_restore --list\` (${db.name}) = ${db.integrity === 'pg_restore --list ok' ? 'ok' : cell(db.integrity).slice(0, 80)}`);
for (const db of r.databases) checks.push(`\`pg_restore --list\` (${db.name}) = ${db.integrity === 'pg_restore --list ok' ? 'ok' : cell(db.integrity).slice(0, 80)}`);
for (const o of r.objects || []) checks.push(o.integrity === 'sha256 ok'
for (const o of r.objects || []) checks.push(o.integrity === 'sha256 ok'
? `object store \`${o.name}\` ${o.blobs} blob(s) sha256 ok`
? `object store \`${o.name}\` ${o.blobs} blob(s) sha256 ok`
- : `object store \`${o.name}\`: ${cell(o.integrity).slice(0, 80)}`);
+ : o.integrity && o.integrity.startsWith('archive only')
+ ? `object store \`${o.name}\`: archive restored (blobs not content-addressed)`
+ : `object store \`${o.name}\`: ${cell(o.integrity).slice(0, 80)}`);
if (r.ready) checks.push(r.ready.ok ? `drill instance \`${new URL(r.ready.url).pathname}\` ${r.ready.status} after ${r.ready.seconds}s` : `drill instance not ready (${r.ready.status || 'no answer'})`);
if (r.ready) checks.push(r.ready.ok ? `drill instance \`${new URL(r.ready.url).pathname}\` ${r.ready.status} after ${r.ready.seconds}s` : `drill instance not ready (${r.ready.status || 'no answer'})`);
const same = r.compare.filter((c) => c.match).map((c) => `\`${c.path}\``);
const same = r.compare.filter((c) => c.match).map((c) => `\`${c.path}\``);
const diff = r.compare.filter((c) => !c.match).map((c) => `\`${c.path}\` differs (${cell(c.detail)})`);
const diff = r.compare.filter((c) => !c.match).map((c) => `\`${c.path}\` differs (${cell(c.detail)})`);
@@ -7,11 +7,16 @@ const { git } = require('../git');
@@ -7,11 +7,16 @@ const { git } = require('../git');
async function statusOne(ctx, svc) {
async function statusOne(ctx, svc) {
const { exec } = ctx;
const { exec } = ctx;
- const row = { service: svc.id, managed: svc.managed, units: [], workers: [], socket: null, sha: null, branch: null, ready: null, protected: null };
+ const row = { service: svc.id, managed: svc.managed, units: [], workers: [], auxiliary: [], socket: null, sha: null, branch: null, ready: null, protected: null };
for (const u of svc.units) {
for (const u of svc.units) {
const st = await systemd.show(exec, u);
const st = await systemd.show(exec, u);
row.units.push({ unit: u, active: st.active, sub: st.sub, restarts: st.restarts });
row.units.push({ unit: u, active: st.active, sub: st.sub, restarts: st.restarts });
}
}
+ // Auxiliary units (Mail's Stalwart) belong to the service but are never restarted with it.
+ for (const a of svc.auxiliaryUnits) {
+ const st = await systemd.show(exec, a.unit);
+ row.auxiliary.push({ unit: a.unit, active: st.active, sub: st.sub, restarts: st.restarts });
+ }
// Worker units are only listed: ovhost never starts, stops or restarts them.
// Worker units are only listed: ovhost never starts, stops or restarts them.
for (const w of svc.workerUnits) {
for (const w of svc.workerUnits) {
try {
try {
@@ -123,6 +123,11 @@ async function validate(ctx, id, opts = {}) {
@@ -123,6 +123,11 @@ async function validate(ctx, id, opts = {}) {
if (!running.length) add('warn', 'units', `no running instance of worker unit ${w}`);
if (!running.length) add('warn', 'units', `no running instance of worker unit ${w}`);
for (const i of list) add('info', 'units', `worker ${i.unit} ${i.active}/${i.sub} (ovhost never starts, stops or restarts it)`);
for (const i of list) add('info', 'units', `worker ${i.unit} ${i.active}/${i.sub} (ovhost never starts, stops or restarts it)`);
}
}
+ for (const a of svc.auxiliaryUnits) {
+ const st = await systemd.show(exec, a.unit);
+ if (st.load !== 'loaded' || !st.fragmentPath) add('error', 'units', `auxiliary ${a.unit} is not loaded (${st.load})`);
+ else add(st.active === 'active' ? 'ok' : 'warn', 'units', `auxiliary ${a.unit} ${st.active}/${st.sub} (restarted with ${svc.id} only when ${a.restartOn.length ? a.restartOn.join(', ') : 'a declared path'} changes)`);
+ }
if (svc.socketUnit) {
if (svc.socketUnit) {
const st = await systemd.show(exec, svc.socketUnit);
const st = await systemd.show(exec, svc.socketUnit);
if (st.load !== 'loaded') add('error', 'units', `${svc.socketUnit} is not loaded (${st.load})`);
if (st.load !== 'loaded') add('error', 'units', `${svc.socketUnit} is not loaded (${st.load})`);
@@ -16,8 +16,12 @@ class InventoryError extends Error {}
@@ -16,8 +16,12 @@ class InventoryError extends Error {}
const ID_RE = /^[a-z][a-z0-9-]{0,39}$/;
const ID_RE = /^[a-z][a-z0-9-]{0,39}$/;
const UNIT_RE = /^[A-Za-z0-9@._-]+\.(service|socket)$/;
const UNIT_RE = /^[A-Za-z0-9@._-]+\.(service|socket)$/;
+// unitSources may also carry a slice (Stalwart's memory cap) and drop-ins; only .service units are restarted.
+const SOURCE_UNIT_RE = /^[A-Za-z0-9@._-]+\.(service|socket|slice)$/;
// A worker unit is a template (`[email protected]`) or a plain service unit.
// A worker unit is a template (`[email protected]`) or a plain service unit.
const WORKER_UNIT_RE = /^[A-Za-z0-9@._-]+\.service$/;
const WORKER_UNIT_RE = /^[A-Za-z0-9@._-]+\.service$/;
+// An auxiliary unit is a plain service unit that belongs to a service but is not restarted with it.
+const AUX_UNIT_RE = /^[A-Za-z0-9@._-]+\.service$/;
const FILE_NAME_RE = /^[A-Za-z0-9._-]+$/;
const FILE_NAME_RE = /^[A-Za-z0-9._-]+$/;
const DRAIN_POLICIES = ['refuse', 'wait', 'report'];
const DRAIN_POLICIES = ['refuse', 'wait', 'report'];
const PROBE_KINDS = ['http-json-count', 'postgresql-count', 'sum'];
const PROBE_KINDS = ['http-json-count', 'postgresql-count', 'sum'];
@@ -144,10 +148,12 @@ function normalisePreflight(raw, where) {
@@ -144,10 +148,12 @@ function normalisePreflight(raw, where) {
}
}
/**
/**
- * `objects`: directories of content-addressed blobs the service owns OUTSIDE its databases, such as
- * OpenVibe.Host's tenant object store (host.example.json: `HOST_STORAGE_DIR`). Each regular file in
- * the directory is named by its own sha256, so `ovhost backup` can archive the tree as one artifact
- * and a drill can check every blob's sha256 after restoring it.
+ * `objects`: directories of blobs the service owns OUTSIDE its databases, such as OpenVibe.Host's
+ * tenant object store (host.example.json: `HOST_STORAGE_DIR`) or Stalwart's message blobs. When the
+ * files are named by their own sha256 (the default), `ovhost backup` archives the tree as one
+ * artifact and a drill checks every blob's sha256 after restoring it. A store that compresses or
+ * encrypts its blobs (Stalwart) cannot pass that check: `verify: false` still archives and restores
+ * the directory, but the drill checks only the archive.
*/
*/
function normaliseObjects(raw, where) {
function normaliseObjects(raw, where) {
if (raw == null) return [];
if (raw == null) return [];
@@ -158,8 +164,38 @@ function normaliseObjects(raw, where) {
@@ -158,8 +164,38 @@ function normaliseObjects(raw, where) {
if (!o || typeof o !== 'object' || Array.isArray(o)) throw new InventoryError(`${at} must be an object`);
if (!o || typeof o !== 'object' || Array.isArray(o)) throw new InventoryError(`${at} must be an object`);
if (!FILE_NAME_RE.test(o.name || '')) throw new InventoryError(`${at}.name must be a plain name`);
if (!FILE_NAME_RE.test(o.name || '')) throw new InventoryError(`${at}.name must be a plain name`);
if (seen.has(o.name)) throw new InventoryError(`${at}.name "${o.name}" is listed twice`);
if (seen.has(o.name)) throw new InventoryError(`${at}.name "${o.name}" is listed twice`);
+ if (o.verify != null && typeof o.verify !== 'boolean') throw new InventoryError(`${at}.verify must be true or false`);
seen.add(o.name);
seen.add(o.name);
- return { name: o.name, path: abs(o.path, `${at}.path`) };
+ return { name: o.name, path: abs(o.path, `${at}.path`), verify: o.verify !== false };
+ });
+}
+
+/**
+ * `auxiliaryUnits`: service units the service depends on but does not run in its own process, and
+ * that ovhost must not restart on every deploy. Each is installed from `unitSources`; `restartOn`
+ * lists checkout paths (config, unit files) whose change is the only reason to restart it. The
+ * deploy restarts an auxiliary unit before the service's own units, and a rollback never does.
+ */
+function normaliseAuxiliaryUnits(raw, where) {
+ if (raw == null) return [];
+ if (!Array.isArray(raw)) throw new InventoryError(`${where}.auxiliaryUnits must be a list of units`);
+ const seen = new Set();
+ return raw.map((a, i) => {
+ const at = `${where}.auxiliaryUnits[${i}]`;
+ const unit = typeof a === 'string' ? a : a && a.unit;
+ if (typeof unit !== 'string' || !AUX_UNIT_RE.test(unit)) throw new InventoryError(`${at}.unit must be a .service unit`);
+ if (seen.has(unit)) throw new InventoryError(`${at}.unit "${unit}" is listed twice`);
+ seen.add(unit);
+ const restartOn = typeof a === 'string' ? [] : (a.restartOn || []);
+ if (!Array.isArray(restartOn)) throw new InventoryError(`${at}.restartOn must be a list of paths inside the checkout`);
+ for (const f of restartOn) rel(f, `${at}.restartOn`);
+ // applyOn: paths whose change a restart does not take (Stalwart reads its rendered configuration, so a change
+ // to the template must be rendered and applied first). ovhost restarts nothing for them and says so.
+ const applyOn = typeof a === 'string' ? [] : (a.applyOn || []);
+ if (!Array.isArray(applyOn)) throw new InventoryError(`${at}.applyOn must be a list of paths inside the checkout`);
+ for (const f of applyOn) rel(f, `${at}.applyOn`);
+ const applyHint = typeof a === 'string' || a.applyHint == null ? null : String(a.applyHint).slice(0, 300);
+ return { unit, restartOn, applyOn, applyHint };
});
});
}
}
@@ -212,6 +248,7 @@ function normaliseService(id, raw, defaults) {
@@ -212,6 +248,7 @@ function normaliseService(id, raw, defaults) {
// OpenRestream's transport workers ([email protected], one instance per release), which
// OpenRestream's transport workers ([email protected], one instance per release), which
// drain on their own. status/validate list their instances read-only.
// drain on their own. status/validate list their instances read-only.
workerUnits: s.workerUnits || [],
workerUnits: s.workerUnits || [],
+ auxiliaryUnits: normaliseAuxiliaryUnits(s.auxiliaryUnits, where),
socketUnit: s.socketUnit || null,
socketUnit: s.socketUnit || null,
unitSources: s.unitSources || {},
unitSources: s.unitSources || {},
envFile: s.envFile ? abs(s.envFile, `${where}.envFile`) : null,
envFile: s.envFile ? abs(s.envFile, `${where}.envFile`) : null,
@@ -283,8 +320,13 @@ function normaliseService(id, raw, defaults) {
@@ -283,8 +320,13 @@ function normaliseService(id, raw, defaults) {
}
}
}
}
}
}
+ for (const a of out.auxiliaryUnits) {
+ if (out.units.includes(a.unit)) throw new InventoryError(`${where}.auxiliaryUnits: "${a.unit}" is already in units; an auxiliary unit is not restarted with the service`);
+ if (out.workerUnits.includes(a.unit)) throw new InventoryError(`${where}.auxiliaryUnits: "${a.unit}" is a worker unit; declare it once`);
+ if (!Object.hasOwn(out.unitSources, a.unit)) throw new InventoryError(`${where}.auxiliaryUnits.${a.unit}: needs a unitSources entry so the deploy can install and track it`);
+ }
for (const [unit, src] of Object.entries(out.unitSources)) {
for (const [unit, src] of Object.entries(out.unitSources)) {
- if (!UNIT_RE.test(unit) && !/^[A-Za-z0-9@._-]+\.(service|socket)\.d\/[A-Za-z0-9._-]+\.conf$/.test(unit)) throw new InventoryError(`${where}.unitSources: "${unit}" is not a unit or drop-in name`);
+ if (!SOURCE_UNIT_RE.test(unit) && !/^[A-Za-z0-9@._-]+\.(service|socket)\.d\/[A-Za-z0-9._-]+\.conf$/.test(unit)) throw new InventoryError(`${where}.unitSources: "${unit}" is not a unit, slice or drop-in name`);
rel(src, `${where}.unitSources.${unit}`);
rel(src, `${where}.unitSources.${unit}`);
}
}
if (out.port !== null && !(Number.isInteger(out.port) && out.port > 0 && out.port < 65536)) throw new InventoryError(`${where}.port must be a TCP port`);
if (out.port !== null && !(Number.isInteger(out.port) && out.port > 0 && out.port < 65536)) throw new InventoryError(`${where}.port must be a TCP port`);
@@ -256,6 +256,9 @@ async function computePlan(ctx, svc, { mode = 'deploy', to = null, fetch = true,
@@ -256,6 +256,9 @@ async function computePlan(ctx, svc, { mode = 'deploy', to = null, fetch = true,
preflight: preflight.describe(svc),
preflight: preflight.describe(svc),
restartNeeded,
restartNeeded,
units: svc.units,
units: svc.units,
+ auxiliaryUnits: svc.auxiliaryUnits.map((a) => a.unit),
+ auxRestart: [],
+ auxRestartNeeded: false,
extraUnits: [],
extraUnits: [],
socketUnit: svc.socketUnit,
socketUnit: svc.socketUnit,
socket,
socket,
@@ -268,6 +271,9 @@ async function computePlan(ctx, svc, { mode = 'deploy', to = null, fetch = true,
@@ -268,6 +271,9 @@ async function computePlan(ctx, svc, { mode = 'deploy', to = null, fetch = true,
installVhosts: false,
installVhosts: false,
vhostsRemoved: [],
vhostsRemoved: [],
nginxNotInstalled: !!(svc.nginx && svc.nginx.repoVhost && changed.includes(svc.nginx.repoVhost)),
nginxNotInstalled: !!(svc.nginx && svc.nginx.repoVhost && changed.includes(svc.nginx.repoVhost)),
+ nginxVhost: null,
+ envFile: svc.envFile,
+ databases: svc.databases.map((d) => d.database),
keep: rel.keep,
keep: rel.keep,
};
};
}
}
@@ -194,6 +194,17 @@ async function computePlan(ctx, svc, { mode = 'deploy', to = null, fetch = true,
@@ -194,6 +194,17 @@ async function computePlan(ctx, svc, { mode = 'deploy', to = null, fetch = true,
}
}
const units = [...svc.units, ...extraUnits];
const units = [...svc.units, ...extraUnits];
const restartNeeded = units.length > 0 && (restart || codeChanged.length > 0);
const restartNeeded = units.length > 0 && (restart || codeChanged.length > 0);
+ // An auxiliary unit (Mail's Stalwart) is installed and listed with the service but is not restarted
+ // on every deploy: only a change to one of its declared paths (its rendered config, its unit or
+ // slice) is a reason. Rollbacks never restart it; the rendered config on disk did not change.
+ const auxRestart = mode === 'deploy'
+ ? svc.auxiliaryUnits.filter((a) => a.restartOn.length && changed.some((f) => matchesPath(f, a.restartOn))).map((a) => a.unit)
+ : [];
+ // A change a restart would not take (a configuration template that must be rendered and applied): restart
+ // nothing, and say it is not live yet.
+ const auxApply = mode === 'deploy'
+ ? svc.auxiliaryUnits.filter((a) => (a.applyOn || []).length && changed.some((f) => matchesPath(f, a.applyOn))).map((a) => ({ unit: a.unit, hint: a.applyHint }))
+ : [];
const unitDrift = [];
const unitDrift = [];
for (const [unit, src] of Object.entries(svc.unitSources)) {
for (const [unit, src] of Object.entries(svc.unitSources)) {
@@ -240,6 +251,10 @@ async function computePlan(ctx, svc, { mode = 'deploy', to = null, fetch = true,
@@ -240,6 +251,10 @@ async function computePlan(ctx, svc, { mode = 'deploy', to = null, fetch = true,
preflight: preflight.describe(svc),
preflight: preflight.describe(svc),
restartNeeded,
restartNeeded,
units,
units,
+ auxiliaryUnits: svc.auxiliaryUnits.map((a) => a.unit),
+ auxRestart,
+ auxRestartNeeded: auxRestart.length > 0,
+ auxApply,
extraUnits,
extraUnits,
skippedJobs,
skippedJobs,
socketUnit: svc.socketUnit,
socketUnit: svc.socketUnit,
@@ -250,9 +265,14 @@ async function computePlan(ctx, svc, { mode = 'deploy', to = null, fetch = true,
@@ -250,9 +265,14 @@ async function computePlan(ctx, svc, { mode = 'deploy', to = null, fetch = true,
drainPolicy: svc.drain.policy,
drainPolicy: svc.drain.policy,
managed: svc.managed,
managed: svc.managed,
installVhosts: !!(svc.nginx && svc.nginx.installOnDeploy),
installVhosts: !!(svc.nginx && svc.nginx.installOnDeploy),
+ nginxVhost: svc.nginx && svc.nginx.installOnDeploy && svc.nginx.repoVhost
+ ? { name: svc.nginx.vhost || path.basename(svc.nginx.repoVhost), source: svc.nginx.repoVhost }
+ : null,
vhostsRemoved,
vhostsRemoved,
nginxNotInstalled: !!(svc.nginx && svc.nginx.repoVhost && !svc.nginx.installOnDeploy && changed.includes(svc.nginx.repoVhost)),
nginxNotInstalled: !!(svc.nginx && svc.nginx.repoVhost && !svc.nginx.installOnDeploy && changed.includes(svc.nginx.repoVhost)),
announceFrom: svc.announce.releaseFiles,
announceFrom: svc.announce.releaseFiles,
+ envFile: svc.envFile,
+ databases: svc.databases.map((d) => d.database),
};
};
}
}
@@ -362,7 +382,9 @@ async function installUnits(ctx, svc, record) {
@@ -362,7 +382,9 @@ async function installUnits(ctx, svc, record) {
}
}
record.unitsInstalled = changed;
record.unitsInstalled = changed;
if (svc.enableUnits) {
if (svc.enableUnits) {
- for (const unit of svc.units) {
+ // Auxiliary units are enabled but never started here: they are already running, or the
+ // operator starts them once (Mail's Stalwart needs its rendered config first).
+ for (const unit of [...svc.units, ...svc.auxiliaryUnits.map((a) => a.unit)]) {
const enabled = await systemd.systemctl(ctx.exec, 'enable', unit);
const enabled = await systemd.systemctl(ctx.exec, 'enable', unit);
if (enabled.code !== 0) throw new Error(`systemctl enable ${unit} failed: ${enabled.stderr.trim()}`);
if (enabled.code !== 0) throw new Error(`systemctl enable ${unit} failed: ${enabled.stderr.trim()}`);
}
}
@@ -546,6 +568,15 @@ async function apply(ctx, svc, mode, opts = {}) {
@@ -546,6 +568,15 @@ async function apply(ctx, svc, mode, opts = {}) {
if (opts.installUnits || svc.installUnits) await installUnits(ctx, svc, record);
if (opts.installUnits || svc.installUnits) await installUnits(ctx, svc, record);
if (plan.backupNeeded && plan.restartNeeded) await backupBeforeRestart(ctx, svc, record);
if (plan.backupNeeded && plan.restartNeeded) await backupBeforeRestart(ctx, svc, record);
if (plan.restartNeeded) await guardSessions(ctx, svc, guardedOpts, record, 'immediately before the restart');
if (plan.restartNeeded) await guardSessions(ctx, svc, guardedOpts, record, 'immediately before the restart');
+ for (const a of plan.auxApply || []) {
+ log(`NOTE: ${a.unit}'s configuration changed and is NOT live until it is applied by hand${a.hint ? `: ${a.hint}` : ''}. ovhost restarts nothing for it.`);
+ (record.auxApplyPending = record.auxApplyPending || []).push(a.unit);
+ }
+ for (const u of plan.auxRestart || []) {
+ log(`restarting auxiliary ${u} (its configuration changed; ${svc.units.join(', ')} is not restarted for this)`);
+ await systemd.restart(ctx.exec, u);
+ (record.auxRestarted = record.auxRestarted || []).push(u);
+ }
} catch (err) {
} catch (err) {
// Nothing has been restarted: put the files back the way the running process expects.
// Nothing has been restarted: put the files back the way the running process expects.
if (record.checkoutMoved) {
if (record.checkoutMoved) {
@@ -560,7 +591,8 @@ async function apply(ctx, svc, mode, opts = {}) {
@@ -560,7 +591,8 @@ async function apply(ctx, svc, mode, opts = {}) {
const announcements = await releaseAnnouncements(ctx, svc);
const announcements = await releaseAnnouncements(ctx, svc);
if (!plan.restartNeeded) {
if (!plan.restartNeeded) {
record.result = mode === 'deploy' ? 'deployed' : 'rolled-back';
record.result = mode === 'deploy' ? 'deployed' : 'rolled-back';
- log(`${plan.units.length ? `only ${svc.noRestartPaths.join(' ')} changed — no restart needed` : 'no units to restart'}`);
+ if (plan.auxRestartNeeded) log(`auxiliary ${plan.auxRestart.join(', ')} restarted; ${svc.units.join(', ')} not restarted`);
+ else log(`${plan.units.length ? `only ${svc.noRestartPaths.join(' ')} changed — no restart needed` : 'no units to restart'}`);
if (svc.ready) {
if (svc.ready) {
const p = await readiness.probe(exec, svc);
const p = await readiness.probe(exec, svc);
record.ready = p.ok;
record.ready = p.ok;
@@ -20,7 +20,7 @@ runTests([
@@ -20,7 +20,7 @@ runTests([
// Every service on the host (2026-10-15): Sites retired, and the fourteen added since the first copy of this file.
// Every service on the host (2026-10-15): Sites retired, and the fourteen added since the first copy of this file.
assert.deepStrictEqual(Object.keys(inv.services), ['network', 'live', 'media', 'tools', 'community', 'events', 'games', 'host', 'billing', 'chat', 'search', 'sources', 'wiki', 'blog', 'tips', 'vip', 'news', 'reviews', 'deals', 'coupons', 'trade', 'codes', 'ai', 'bot', 'openre',
assert.deepStrictEqual(Object.keys(inv.services), ['network', 'live', 'media', 'tools', 'community', 'events', 'games', 'host', 'billing', 'chat', 'search', 'sources', 'wiki', 'blog', 'tips', 'vip', 'news', 'reviews', 'deals', 'coupons', 'trade', 'codes', 'ai', 'bot', 'openre',
'actor', 'food', 'help', 'work', 'quest', 'rent', 'media-hub', 'inventory', 'play', 'zone', 'watch', 'run', 'services', 'space',
'actor', 'food', 'help', 'work', 'quest', 'rent', 'media-hub', 'inventory', 'play', 'zone', 'watch', 'run', 'services', 'space',
- 'design', 'exchange', 'report', 'vision', 'fun', 'shoes', 'bench', 'cam', 'domains', 'meme', 'surf', 'today', 'tours']);
+ 'design', 'exchange', 'report', 'vision', 'fun', 'shoes', 'bench', 'cam', 'domains', 'meme', 'surf', 'today', 'tours', 'mail']);
assert.ok(Object.values(inv.services).filter((s) => s.databases.length).every((s) => s.drill), 'every service with a database has a restore drill');
assert.ok(Object.values(inv.services).filter((s) => s.databases.length).every((s) => s.drill), 'every service with a database has a restore drill');
assert.strictEqual(inv.services.live.socketUnit, 'openvibe-live.socket');
assert.strictEqual(inv.services.live.socketUnit, 'openvibe-live.socket');
assert.ok(!inv.services.live.units.includes('openvibe-live.socket'));
assert.ok(!inv.services.live.units.includes('openvibe-live.socket'));
@@ -42,8 +42,22 @@ runTests([
@@ -42,8 +42,22 @@ runTests([
assert.deepStrictEqual([inv.services.bot.nginx.vhost, inv.services.bot.nginx.repoVhost, inv.services.bot.nginx.installOnDeploy], ['openvibe.bot.conf', 'deploy/nginx/openvibe.bot.conf', true]);
assert.deepStrictEqual([inv.services.bot.nginx.vhost, inv.services.bot.nginx.repoVhost, inv.services.bot.nginx.installOnDeploy], ['openvibe.bot.conf', 'deploy/nginx/openvibe.bot.conf', true]);
assert.strictEqual(inv.services.bot.installUnits, true);
assert.strictEqual(inv.services.bot.installUnits, true);
assert.strictEqual(inv.services.bot.enableUnits, true);
assert.strictEqual(inv.services.bot.enableUnits, true);
+ // Mail: the Node API on loopback, Stalwart as an auxiliary unit that a deploy does not restart,
+ // the database (Stalwart's data store too) and the message blobs outside the database.
+ assert.deepStrictEqual([inv.services.mail.port, inv.services.mail.envFile, inv.services.mail.ready.url], [5250, '/etc/openvibe/mail.env', 'http://127.0.0.1:5250/api/ready']);
+ assert.deepStrictEqual(inv.services.mail.units, ['openvibe-mail.service']);
+ assert.deepStrictEqual(inv.services.mail.auxiliaryUnits.map(({ unit, restartOn, applyOn }) => ({ unit, restartOn, applyOn })),
+ [{ unit: 'openvibe-stalwart.service', restartOn: ['deploy/systemd/openvibe-stalwart.service', 'deploy/systemd/openvibe-stalwart.slice'], applyOn: ['deploy/stalwart/'] }]);
+ assert.match(inv.services.mail.auxiliaryUnits[0].applyHint, /render and apply/);
+ assert.strictEqual(inv.services.mail.unitSources['openvibe-stalwart.slice'], 'deploy/systemd/openvibe-stalwart.slice');
+ assert.deepStrictEqual([inv.services.mail.databases[0].database, inv.services.mail.databases[0].role], ['ov_mail', 'ov_mail']);
+ assert.deepStrictEqual(inv.services.mail.objects, [{ name: 'stalwart-blobs', path: '/var/lib/openvibe-stalwart/blobs', verify: false }]);
+ assert.deepStrictEqual([inv.services.mail.nginx.vhost, inv.services.mail.nginx.repoVhost, inv.services.mail.nginx.installOnDeploy], ['openvibe.email.conf', 'deploy/nginx/openvibe.email.conf', true]);
+ assert.strictEqual(inv.services.mail.drill.supported, false);
+ assert.ok(inv.services.mail.lifecycle.shutdown.deadlineSeconds <= 20, 'within openvibe-mail.service TimeoutStopSec');
const text = JSON.stringify(raw);
const text = JSON.stringify(raw);
- assert.ok(!/(sk_|ghp_|BEGIN [A-Z ]*PRIVATE KEY|password\s*[:=]\s*\S)/i.test(text), 'no secret values in the example');
+ // Token-shaped values only: MAIL_MASK_DOMAIN is a name, not a secret (the old /i "sk_" caught it).
+ assert.ok(!/(\bsk_[A-Za-z0-9]{8,}|\bghp_[A-Za-z0-9]{8,}|BEGIN [A-Z ]*PRIVATE KEY|password\s*[:=]\s*\S)/.test(text), 'no secret values in the example');
}),
}),
test('the released service manifest is valid against the contracts schema', () => {
test('the released service manifest is valid against the contracts schema', () => {
new file mode 100644
new file mode 100644
@@ -0,0 +1,155 @@
@@ -0,0 +1,155 @@
+'use strict';
+/**
+ * OpenVibe.Mail in the host inventory: the Node API on 5250, Stalwart as an auxiliary unit that a
+ * normal Mail deploy never restarts, and the message blobs archived with the database.
+ */
+const assert = require('assert');
+const fs = require('fs');
+const path = require('path');
+const { createFakeHost } = require('./fake-host');
+const { main } = require('../lib/cli');
+const { normalise } = require('../lib/inventory');
+
+const STATE = '/var/lib/openvibe-host';
+
+const MAIL_FILES = {
+ 'package.json': JSON.stringify({ name: 'openvibe-mail', version: '0.1.0', dependencies: { express: '^4.21.2' } }, null, 2),
+ 'package-lock.json': '{"lockfileVersion":3,"packages":{}}',
+ 'server/index.js': 'console.log(1);',
+ '.env.example': 'DATABASE_URL=\nDATABASE_DIRECT_URL=\n',
+ 'deploy/systemd/openvibe-mail.service': '[Service]\nExecStart=/usr/bin/node server/index.js\nTimeoutStopSec=20\n',
+ 'deploy/systemd/openvibe-stalwart.service': '[Service]\nExecStart=/opt/openvibe.stalwart/stalwart --config /etc/openvibe/stalwart/config.json\n',
+ 'deploy/systemd/openvibe-stalwart.slice': '[Slice]\nMemoryMax=1G\n',
+ 'deploy/stalwart/config.toml': '# Stalwart configuration\n',
+ 'deploy/nginx/openvibe.email.conf': 'server { listen 443; }\n',
+};
+
+function mailHost() {
+ const host = createFakeHost();
+ const example = JSON.parse(fs.readFileSync(path.join(__dirname, '..', 'host.example.json'), 'utf8'));
+ const doc = {
+ host: 'fake-host',
+ stateDir: STATE,
+ backupDir: '/var/backups/openvibe',
+ defaults: { owner: 'ubuntu', branch: 'main' },
+ services: { mail: example.services.mail },
+ };
+ host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });
+ host.inv = normalise(doc);
+ const svc = host.inv.services.mail;
+ for (const db of svc.databases) host.pgDatabases.add(db.database);
+ host.pgBackups = [{ type: 'diff', timestamp: { stop: Math.floor(host.exec.now() / 1000) - 3600 } }];
+ host.ensureDir('/etc/nginx/sites-available');
+ host.ensureDir('/etc/nginx/sites-enabled');
+ host.badShas = new Set();
+ host.cli = async (...argv) => {
+ const lines = [];
+ const code = await main(argv, { exec: host.exec, out: (s) => lines.push(s), env: {} });
+ return { code, out: lines.join('\n') };
+ };
+
+ const repo = host.createRepo(svc.repo, { owner: 'ubuntu' });
+ const sha = repo.commit(MAIL_FILES, { message: 'initial' });
+ repo.publish(sha);
+ repo.checkout(sha);
+ for (const dep of Object.keys(JSON.parse(MAIL_FILES['package.json']).dependencies)) {
+ host.put(path.join(svc.repo, 'node_modules', dep, 'package.json'), JSON.stringify({ name: dep }), { owner: 'ubuntu' });
+ }
+ for (const u of [...svc.units, ...svc.auxiliaryUnits.map((a) => a.unit)]) host.addUnit(u);
+ for (const [unit, src] of Object.entries(svc.unitSources)) host.put(path.join('/etc/systemd/system', unit), MAIL_FILES[src]);
+ host.put(svc.envFile, svc.env.required.map((n) => `${n}=test`).join('\n'), { mode: 0o600 });
+ host.http.set(svc.ready.url, () => ({ status: 200, body: { status: 'ok' } }));
+ host.push = (changes, message = 'change') => { const s = repo.commit(changes, { message }); repo.publish(s); return s; };
+ return host;
+}
+
+function lastRecord(host, id) {
+ const lines = host.read(`${STATE}/backups/${id}.jsonl`).trim().split('\n');
+ return JSON.parse(lines[lines.length - 1]);
+}
+
+(async () => {
+ let failed = 0;
+ const atest = async (name, fn) => {
+ try { await fn(); console.log(` ✓ ${name}`); } catch (err) { failed += 1; console.log(` ✗ ${name}\n${err.stack || err}`); }
+ };
+
+ await atest('plan names the Mail unit, Stalwart auxiliary unit, env, database and vhost', async () => {
+ const host = mailHost();
+ host.push({ 'server/index.js': 'console.log(2);' }, 'server change');
+ const r = await host.cli('plan', 'mail');
+ assert.strictEqual(r.code, 0, r.out);
+ assert.match(r.out, /^ repo\s+\/opt\/openvibe\.mail$/m);
+ assert.match(r.out, /^ env\s+\/etc\/openvibe\/mail\.env$/m);
+ assert.match(r.out, /^ databases\s+ov_mail$/m);
+ assert.match(r.out, /^ restart\s+yes: openvibe-mail\.service$/m);
+ assert.match(r.out, /^ auxiliary\s+openvibe-stalwart\.service \(installed with the service; not restarted unless its config changes\)$/m);
+ assert.match(r.out, /^ nginx\s+openvibe\.email\.conf from deploy\/nginx\/openvibe\.email\.conf installed \(nginx -t, reload\) on deploy$/m);
+ assert.doesNotMatch(r.out, /^ restart\s+yes:.*openvibe-stalwart/m);
+ });
+
+ await atest('a Mail code deploy restarts openvibe-mail.service and never Stalwart', async () => {
+ const host = mailHost();
+ host.push({ 'server/index.js': 'console.log(2);' }, 'server change');
+ const r = await host.cli('deploy', 'mail');
+ assert.strictEqual(r.code, 0, r.out);
+ assert.deepStrictEqual(host.restarts(), ['openvibe-mail.service']);
+ assert.strictEqual(host.units.get('openvibe-stalwart.service').restarts, 0);
+ assert.ok(host.read('/etc/nginx/sites-available/openvibe.email.conf'), 'the repo vhost was installed');
+ assert.ok(host.files.has('/etc/nginx/sites-enabled/openvibe.email.conf'), 'the vhost was linked into sites-enabled');
+ });
+
+ await atest('a Stalwart template change restarts nothing and says it is not live until applied', async () => {
+ const host = mailHost();
+ host.push({ 'deploy/stalwart/config.toml': '# Stalwart configuration, changed\n' }, 'mail server change');
+ const r = await host.cli('deploy', 'mail');
+ assert.strictEqual(r.code, 0, r.out);
+ assert.deepStrictEqual(host.restarts(), [], 'a restart would keep the old configuration, so nothing restarts');
+ assert.match(r.out, /openvibe-stalwart\.service's configuration changed and is NOT live until it is applied by hand: render and apply/);
+ const [rec] = (host.read(`${STATE}/releases/mail.jsonl`) || '').trim().split('\n').map((l) => JSON.parse(l));
+ assert.deepStrictEqual(rec.auxApplyPending, ['openvibe-stalwart.service']);
+ assert.strictEqual(rec.restarted, false);
+ });
+
+ await atest('a Stalwart unit change restarts only the auxiliary unit', async () => {
+ const host = mailHost();
+ host.push({ 'deploy/systemd/openvibe-stalwart.slice': '[Slice]\nMemoryMax=1G\nCPUQuota=100%\n# changed\n' }, 'slice change');
+ const r = await host.cli('deploy', 'mail');
+ assert.strictEqual(r.code, 0, r.out);
+ assert.deepStrictEqual(host.restarts(), ['openvibe-stalwart.service']);
+ assert.strictEqual(host.units.get('openvibe-mail.service').restarts, 0);
+ assert.match(r.out, /restarting auxiliary openvibe-stalwart\.service/);
+ const [rec] = (host.read(`${STATE}/releases/mail.jsonl`) || '').trim().split('\n').map((l) => JSON.parse(l));
+ assert.deepStrictEqual(rec.auxRestarted, ['openvibe-stalwart.service']);
+ });
+
+ await atest('a docs-only change restarts nothing', async () => {
+ const host = mailHost();
+ host.push({ 'docs/INSTALL.md': 'changed\n' }, 'docs change');
+ const r = await host.cli('deploy', 'mail');
+ assert.strictEqual(r.code, 0, r.out);
+ assert.deepStrictEqual(host.restarts(), []);
+ });
+
+ await atest('backup archives ov_mail and the Stalwart blob directory together', async () => {
+ const host = mailHost();
+ host.put('/var/lib/openvibe-stalwart/blobs/aa/bb/0123456789abcdef', 'compressed message bytes', { owner: 'openvibe', mode: 0o600 });
+ const r = await host.cli('backup', 'mail', '--logical');
+ assert.strictEqual(r.code, 0, r.out);
+ const rec = lastRecord(host, 'mail');
+ assert.strictEqual(rec.ok, true);
+ assert.strictEqual(host.dumps[0].database, 'ov_mail');
+ const names = rec.files.map((f) => f.file).sort();
+ assert.deepStrictEqual(names, ['mail.dump', 'stalwart-blobs.tar.gz']);
+ assert.ok(host.files.has(`${rec.dir}/stalwart-blobs.tar.gz`));
+ });
+
+ await atest('a non-content-addressed object store must say verify:false', () => {
+ const host = mailHost();
+ assert.strictEqual(normalise({ services: { mail: host.inv.services.mail } }).services.mail.objects[0].verify, false);
+ assert.throws(() => normalise({ services: { x: { repo: '/opt/x', owner: 'ubuntu', objects: [{ name: 'b', path: '/var/lib/b', verify: 'yes' }] } } }), /verify must be true or false/);
+ });
+
+ if (failed) { console.error(`${failed} failed`); process.exit(1); }
+ console.log('all mail inventory tests passed');
+})().catch((err) => { console.error(err); process.exit(1); });