openvibers/openvibe.host Commits
Back up declared tenant object directories with the databases, offsite included, and verify every blob's sha256 in a restore drill (#23)
Changes
@@@ -404,11 +404,11 @@ The service runs on the host (loopback, since 2026-09-23) and the pending tenant### Not done yet (Stage B)- Uploads are held in memory while they are validated (bounded by `HOST_MAX_UPLOAD_BYTES` and `HOST_MAX_UNPACKED_BYTES`), so the service needs that much headroom per concurrent upload.--- Objects live on the service's local disk, not in OpenVibe.Media; there is no replication and no backup beyond `ovhost backup host`. The database (`ov_host` on the host's data role, ADR-035; schema in [migrations/](migrations/)) is backed up with the others by pgBackRest.++- Objects live on the service's local disk, not in OpenVibe.Media; there is no replication. `ovhost backup host` archives `/var/lib/openvibe-host-api/objects` (the inventory's `objects` entry) alongside `ov_host`, uploads the encrypted archive off-host, and a restore drill extracts it and checks every blob's sha256. The database (`ov_host` on the host's data role, ADR-035; schema in [migrations/](migrations/)) is backed up with the others by pgBackRest.- Certificates for custom domains are issued and installed by `ovhost certs renew --install` (step 8) on the `openvibe-certs.timer`; certbot stays outside the Host service and no key file is read.- Projects are created in Host. When Network has projects (ADR-014), Host should accept only Network project ids and read membership from Network.- `openvibe.host` is not on the Public Suffix List, so a tenant page can still set `Domain=openvibe.host` cookies (a cookie bomb breaks the dashboard and other tenant sites for that visitor). The protections above do not depend on it. [docs/threat-review.md §5](docs/threat-review.md#5-the-public-suffix-list-question) records the decision (launch without it) and the recommended follow-up (dashboard off the tenant zone, then list the zone).--- Tenant objects are not in `ovhost backup` (only `host.db`).++- Tenant objects are in `ovhost backup`: the inventory declares them as `objects` (`/var/lib/openvibe-host-api/objects`), the backup archives the tree, the off-host copy encrypts it, and a drill extracts it and checks every blob's sha256.- Git deploys are Phase 1 only (see [Git deploys](#git-deploys-phase-1)): the project's own CI builds and posts the output. Host never clones, fetches, holds a repository credential or builds; in-Host builds belong to Run (plan T14). There is no webhook: a push reaches Host only through the CI's call.- Tenant sites publish their own sitemap/robots/feed; when a deploy ships no `sitemap.xml` or `robots.txt`, Host generates one (the manifest's HTML pages on the host the request came in on, so a verified custom domain gets its own `sitemap.xml`; sandbox projects get `Disallow: /` and are `noindex`). A tenant's own file at either path always wins. The dashboard host serves `/robots.txt` and `/sitemap.xml` (front page and legal pages).- The Codes portal (Wave 20) does not exist, so there is no public developer onboarding for Host yet.@@@ -14,7 +14,7 @@ Restore drills (a second instance started from a backup and compared with produc## What is backed up`ovhost backup --all` backs up every service in the inventory (`/etc/openvibe/host.json`) that has--a `databases` list. `databases[]` items are one of two **engines** ([db-inventory.md](db-inventory.md)):++a `databases` or `objects` list. `databases[]` items are one of two **engines** ([db-inventory.md](db-inventory.md)):SQLite, a file the service user's worker copies; and PostgreSQL, a database in the host's cluster(ADR-035) that ovhost verifies through pgBackRest and periodically dumps. With `host.example.json`:@@@ -34,6 +34,13 @@ services](#postgresql-services) covers what that means.A database that does not exist yet (a service that is not deployed) is reported as `skipped` anddoes not fail the run. Adding a database to the inventory is all it takes to include it.++**Object directories.** A service may also declare `objects`: a list of `{ name, path }` directories++whose files are content-addressed, each named by its own sha256. OpenVibe.Host's tenant object store++(`HOST_STORAGE_DIR=/var/lib/openvibe-host-api/objects`) is the first. `ovhost backup` archives each++directory as `<name>.tar.gz` (root:root 0600, like every other copy); the off-host code encrypts and++uploads it with the rest, and a [drill](restore-drills.md) extracts it into its own scratch directory++and checks every blob's sha256.++## One runFor each service, one after another:@@@ -448,7 +455,7 @@ Do this at least once after installing, and again after any change to the key or| Command | What it does | Exit ||---|---|---|--| `ovhost backup --all [--offsite] [--logical] [--no-prune] [--keep-daily n] [--keep-weekly n]` | back up every service with databases (verify pgBackRest, dump SQLite, dump PostgreSQL when due or `--logical`); prune; summary; optionally upload | 0 all ok · 1 lock/usage · 2 any failure |++| `ovhost backup --all [--offsite] [--logical] [--no-prune] [--keep-daily n] [--keep-weekly n]` | back up every service with databases or object directories (verify pgBackRest, dump SQLite, dump PostgreSQL when due or `--logical`, archive object directories); prune; summary; optionally upload | 0 all ok · 1 lock/usage · 2 any failure || `ovhost offsite push [--run <run>]` | upload a run (default: the latest) again, for example after a failed upload | 0 · 1 config · 2 failure || `ovhost offsite list [<service>] [--from-host <h>]` | runs off-host, newest first | 0 · 1 config · 2 bucket error || `ovhost offsite check` | config, key and bucket access; uploads nothing | 0 · 1 config · 2 bucket error |@@@ -92,6 +92,9 @@ block (see `host.example.json`):`PRAGMA integrity_check`, run as the service user, must return `ok` for each copy;otherwise the drill fails before anything starts. A PostgreSQL service restores its `.dump` into ascratch database instead — see [A PostgreSQL drill](#a-postgresql-drill).++ Every `objects` directory the service declares is restored as well: its archive is extracted into++ `<tmp>/<name>`, owned by the service user and never the live directory, and every blob is checked++ against the sha256 in its file name.2. **Start.** A second instance starts from the production checkout through `systemd-run`, as theservice user. It uses the production unit's `ExecStart` and `WorkingDirectory`, the productionunit's non-secret `Environment=` values and the production env file. After that it loads@@@ -184,7 +187,7 @@ Supported with declared overrides (the table above records which have passed on- network- community- events--- host (the Host API: `host.db` only, counts only)++- host (the Host API: the `ov_host` database and the tenant object store, whose blobs are sha256-checked; counts only)- billing- chat- search@@@ -285,6 +285,6 @@ the history of any site.| Unverified or lapsed custom domains reach nginx's default server | medium (confusion, mis-served content on a stranger's domain) | operator: catch-all `default_server` (docs/launch.md step 0) || Cookie bomb across tenants and the dashboard (no PSL) | low (per-visitor nuisance; takedown exists) | owner decision (§5) || No abuse mailbox/owner | medium (process, not code) | owner |--| Objects not backed up | medium for tenants (alpha, documented) | follow-up: add the object store to the backup run |++| Objects not backed up | medium for tenants (alpha, documented) | fixed: the inventory's `objects` entry archives the store, the off-host copy encrypts it, and a drill restores it and checks every blob's sha256 || Decompressing an upload blocked the event loop (`gunzipSync`) | done: async `zlib.gunzip` on the libuv threadpool (`server/artifacts/archive.js`), same `maxOutputLength` cap | done || Takedown routes not in released `openvibe-contracts` `host.site.manage.implementedBy` | none at runtime (the contract check passes) | Contracts: next release (docs/launch.md) |@@@ -940,7 +940,7 @@}},"host": {-- "_note": "OpenVibe.Host Stage B (tenant static hosting): the Host API service on port 4910. Not deployed as of 22 Sep 2026. Its content-addressed objects live in /var/lib/openvibe-host-api, which is NOT ovhost's own stateDir (/var/lib/openvibe-host); its database is the PostgreSQL database ov_host. Tenant vhosts: `ovhost nginx tenants host`.",++ "_note": "OpenVibe.Host Stage B (tenant static hosting): the Host API service on port 4910. Not deployed as of 22 Sep 2026. Its content-addressed objects live in /var/lib/openvibe-host-api/objects (`HOST_STORAGE_DIR`), which is NOT ovhost's own stateDir (/var/lib/openvibe-host); its database is the PostgreSQL database ov_host. Tenant vhosts: `ovhost nginx tenants host`.","repo": "/opt/openvibe.host","units": ["openvibe-host.service"@@@ -976,6 +976,12 @@"database": "ov_host"}],++ "objects": [++ {++ "name": "objects",++ "path": "/var/lib/openvibe-host-api/objects"++ }++ ],"nginx": {"tenants": {"sitesDomain": "openvibe.host",@@@ -1036,7 +1042,7 @@"table": "host_domains"}],-- "_notes": "Restores the ov_host database only: `ovhost backup host` does not copy the content-addressed objects, so tenant content cannot be compared (every /api/v1 read needs auth). HOST_WORKER=off stops DNS rechecks and the object sweep."++ "_notes": "Restores the ov_host database and the archived `objects` directory into {tmp}/objects (HOST_STORAGE_DIR), checking every blob's sha256; tenant content is still not compared (every /api/v1 read needs auth). HOST_WORKER=off stops DNS rechecks and the object sweep."}},"billing": {@@@ -1,7 +1,8 @@'use strict';/*** ovhost backup <service>: an online copy of every database the inventory declares for the service,-- * into <backupDir>/<service>/<YYYYMMDD-HHMMSS>/<name>.<ext>, recorded in++ * and a tar archive of every content-addressed object directory it declares, into++ * <backupDir>/<service>/<YYYYMMDD-HHMMSS>/<name>.<ext>, recorded in* <stateDir>/backups/<service>.jsonl. SQLite is copied by a worker running as the service user (root* never opens a service database — see sqlite-worker.js) and checked with PRAGMA quick_check.* PostgreSQL is not dumped every night: pgBackRest holds the cluster's full + differential backups@@@ -30,10 +31,11 @@* throws a BackupError carrying the record. A SQLite backup that found none of its databases is* recorded as ok: false, empty: true and leaves no directory behind; PostgreSQL never sets empty.*-- * ovhost backup --all (backupAll): every service that declares databases, one after another. A-- * failing service never stops the others. Afterwards each service's backups are pruned to the last-- * 7 daily and 4 weekly good ones (lib/retention.js), and a JSON summary of the run is written to-- * <stateDir>/backup-runs/<run>.json. With --offsite the run is then uploaded (lib/offsite.js).++ * ovhost backup --all (backupAll): every service that declares databases or object directories, one++ * after another. A failing service never stops the others. Afterwards each service's backups are++ * pruned to the last 7 daily and 4 weekly good ones (lib/retention.js), and a JSON summary of the run++ * is written to <stateDir>/backup-runs/<run>.json. With --offsite the run is then uploaded++ * (lib/offsite.js).*/const path = require('path');const { service } = require('../inventory');@@@ -46,6 +48,19 @@ const DAY_MS = 86400000;// differentials: it exists for the off-site copy and per-service drills, not for durability.const LOGICAL_DUMP_INTERVAL_DAYS = 7;const ARCHIVER_STATE = 'pgbackrest-archiver.json';++// Room a tar archive of an object directory needs beyond `du -sb` of its source: per-file tar headers++// and a little gzip overhead for already-compressed blobs.++const ARCHIVE_HEADROOM = 1024 * 1024;++// Host stages an upload at <store root>/tmp/<random>.part and renames it into place, so the store is++// never quiescent while it is tarred. These warnings are not failures: the scratch is excluded too.++const BENIGN_TAR_RE = /file changed as we read it|file removed before we read it|Cannot stat: No such file or directory/i;++++/** GNU tar exits 1 when a file changed or vanished mid-read. That is not a failed archive when every++ * warning is one of those; any other stderr line means the run failed. */++function tarWarningsOnly(stderr) {++ const lines = String(stderr || '').split('\n').map((l) => l.trim()).filter(Boolean);++ return lines.length > 0 && lines.every((l) => BENIGN_TAR_RE.test(l));++}class BackupError extends Error {constructor(message, record) { super(message); this.record = record; }@@@ -130,7 +145,7 @@ async function lastGoodDumpAt(exec, inv, id) {async function backup(ctx, id, { reason = 'manual', logical = false } = {}) {const { exec, inv, log } = ctx;const svc = service(inv, id);-- if (!svc.databases.length) throw new Error(`${id} declares no databases`);++ if (!svc.databases.length && !svc.objects.length) throw new Error(`${id} declares no databases or object directories`);const postgres = svc.databases.filter((d) => d.engine === 'postgresql');const hasPostgres = postgres.length > 0;const engine = hasPostgres ? 'postgresql' : 'sqlite';@@@ -156,8 +171,9 @@ async function backup(ctx, id, { reason = 'manual', logical = false } = {}) {const at = last === null ? NaN : Date.parse(last);if (logical || !Number.isFinite(at) || exec.now() - at > LOGICAL_DUMP_INTERVAL_DAYS * DAY_MS) dump.push(...postgres);}-- // A verification-only run (PostgreSQL, no dump due, no SQLite database) leaves nothing on disk.-- if (!dump.length && !svc.databases.some((d) => d.engine === 'sqlite')) {++ // A verification-only run (PostgreSQL, no dump due, no SQLite database, no objects) leaves++ // nothing on disk.++ if (!dump.length && !svc.databases.some((d) => d.engine === 'sqlite') && !svc.objects.length) {const record = { service: id, at: new Date(exec.now()).toISOString(), reason, engine, ok: true, files: [], verified };await exec.appendFile(path.join(inv.stateDir, 'backups', `${id}.jsonl`), `${JSON.stringify(record)}\n`);return record;@@@ -211,9 +227,46 @@ async function backup(ctx, id, { reason = 'manual', logical = false } = {}) {}files.push({ name: db.name, source: db.database, dest: path.join(dir, file), file, check: 'pg_dump -Fc', seconds: Math.round((exec.now() - t0) / 1000) });}-- // Take the staging directory back first: after this the service user (or postgres) can no-- // longer reach anything inside it, so what is checked below cannot change underneath.++ // Take the staging directory back FIRST: while the service user (or postgres) owns it, it could++ // plant a symlink at the path root is about to write the object archive through, and root's++ // `tar -czf` would follow it. Once root:root 0700, neither can reach it, and everything written++ // below is checked before the move.await lockDown(exec, stage, 0o700);++ // Content-addressed object directories: one tar archive each, created by root AFTER the stage++ // is root-locked. The drill extracts this archive into its own scratch directory and checks++ // every blob's sha256; nothing here needs to open a blob individually.++ for (const obj of svc.objects) {++ const file = `${obj.name}.tar.gz`;++ const archive = path.join(stage, file);++ if (!(await exec.stat(obj.path))) { files.push({ name: obj.name, source: obj.path, skipped: 'not found' }); log(`skip ${obj.path}: not found`); continue; }++ const t0 = exec.now();++ // `du -sb` of the source: the uncompressed size the drill must find room for, and the++ // figure the staging space check below is made against.++ const du = await exec.run('du', ['-sb', '--', obj.path], { privileged: true });++ const sourceBytes = du.code === 0 ? Number(String(du.stdout).trim().split(/\s+/)[0]) : null;++ if (Number.isFinite(sourceBytes)) {++ const free = (await exec.statfs(inv.backupStagingDir)).free;++ const needed = sourceBytes + ARCHIVE_HEADROOM;++ if (free < needed) {++ files.push({ name: obj.name, source: obj.path, dest: path.join(dir, file), error: `not enough free space under ${inv.backupStagingDir}: ${free} bytes free, need ${needed} for ${obj.name}` });++ log(`FAILED ${obj.path}: not enough free space under ${inv.backupStagingDir} (${free} < ${needed})`);++ continue;++ }++ }++ // Clear whatever might sit at the archive's path — a link planted while the service user++ // owned the stage would survive lockDown — before root's tar opens it for writing.++ await exec.run('rm', ['-rf', '--', archive], { privileged: true });++ log(`backing up object directory ${obj.path} → ${path.join(dir, file)}`);++ // Host's transient upload scratch lives at <root>/tmp/*.part; exclude it, and tolerate a++ // tar exit of 1 whose only warnings are a file that changed or vanished mid-read.++ const r = await exec.run('tar', ['-czf', archive, '-C', obj.path, '--exclude=./tmp', '--exclude=./tmp/*', '.'], { privileged: true });++ if (r.code !== 0 && !(r.code === 1 && tarWarningsOnly(r.stderr))) {++ files.push({ name: obj.name, source: obj.path, dest: path.join(dir, file), error: `tar: ${(r.stderr || '').trim().slice(0, 200)}` });++ log(`FAILED ${obj.path}: ${(r.stderr || '').trim().slice(0, 200)}`);++ continue;++ }++ files.push({ name: obj.name, source: obj.path, dest: path.join(dir, file), file, kind: 'objects', sourceBytes, check: 'object archive', seconds: Math.round((exec.now() - t0) / 1000) });++ }for (const f of files.filter((x) => x.file && !x.error)) {const p = path.join(stage, f.file);const fst = await exec.stat(p);@@@ -299,7 +352,7 @@ async function backupAll(ctx, { daily = 7, weekly = 4, prune: doPrune = true, ofconst file = path.join(inv.stateDir, 'backup-runs', `${run}.json`);try {for (const svc of Object.values(inv.services)) {-- if (!svc.databases.length) continue;++ if (!svc.databases.length && !svc.objects.length) continue;const entry = { service: svc.id, status: 'ok', dir: null, files: [], error: null, pruned: [] };summary.services.push(entry);try {@@@ -12,6 +12,9 @@* postgres, verify it with `pg_restore --list`, then create a scratch database and* a login role `ov_<id>_drill_<stamp>` and `pg_restore --no-owner --role=…` the* archive into the scratch database. Both are dropped again in a finally.++ * Every object directory the service declares is also restored: its archive is extracted into++ * <tmp>/<name> (owned by the service user, never the live directory) and every blob is checked++ * against the sha256 in its file name.* 2. Start a second instance from the production checkout, as the service user, through* systemd-run. It uses the production unit's ExecStart and WorkingDirectory, the production* env file, and a second env file with the drill's overrides: its port, the restored database@@@ -43,6 +46,8 @@* allows loopback).*/const path = require('path');++const crypto = require('crypto');++const { pipeline } = require('stream/promises');const { service } = require('../inventory');const envfile = require('../envfile');const systemd = require('../systemd');@@@ -147,12 +152,83 @@ async function countRows(exec, db, table, as) {return n;}++// A content-addressed blob is named by its own sha256: that name is the whole integrity check.++const BLOB_RE = /^[0-9a-f]{64}$/;++++/** Every non-directory entry under `dir`, recursively (regular files and anything else tar restored). */++async function walkFiles(exec, dir) {++ const out = [];++ for (const e of (await exec.readdir(dir)) || []) {++ const p = path.join(dir, e.name);++ const st = await exec.stat(p);++ if (!st) continue;++ // Never follow a symlink: `exec.stat` follows it, so a restored member `x -> /` would otherwise++ // be recursed into and walk (and hash) files outside the scratch directory as root. It is++ // returned as a leaf so verifyObjectBlobs refuses it.++ if (st.isDir && !st.isSymlink) out.push(...await walkFiles(exec, p));++ else out.push(p);++ }++ return out;++}++++/** The sha256 of a file, streamed so a large blob is never held in memory. */++async function sha256Stream(exec, p) {++ const hash = crypto.createHash('sha256');++ await pipeline(exec.readStream(p), hash);++ return hash.digest('hex');++}++++/**++ * Check a restored object directory: every regular file is a blob named by its own sha256. A file++ * whose name is not a sha256 is not a blob (Host keeps transient write scratch under its store++ * root) and is counted as skipped, never verified. Any symlink or special file (device, fifo,++ * socket) in the restored tree is refused whatever its name, as is a blob-named entry that is not a++ * plain file, as in a database copy. `entry` gets the counts and the sha256 mismatch list.++ */++async function verifyObjectBlobs(exec, dir, entry) {++ const bad = [];++ for (const p of await walkFiles(exec, dir)) {++ const st = await exec.stat(p);++ if (!st) continue;++ const name = path.basename(p);++ // walkFiles returns symlinks as leaves; a special file is neither a file nor a directory.++ if (st.isSymlink || !st.isFile) { bad.push(`${path.relative(dir, p)} (symlink or special file)`); continue; }++ if (!BLOB_RE.test(name)) { entry.skipped += 1; entry.bytes += st.size; continue; }++ if (st.nlink !== 1) { bad.push(`${path.relative(dir, p)} (not a plain file)`); continue; }++ entry.bytes += st.size;++ const hash = await sha256Stream(exec, p);++ if (hash !== name) { bad.push(`${path.relative(dir, p)} (sha256 ${hash})`); continue; }++ entry.blobs += 1;++ }++ entry.mismatches = bad.length;++ if (bad.length) throw new Error(`${bad.length} blob(s) do not match their name: ${bad.slice(0, 3).join(', ')}${bad.length > 3 ? ', …' : ''}`);++}++++/**++ * A `tar -tvzf` listing line: permissions, owner/group, size, date, time, name. Only a regular file++ * ('-') or a directory ('d') may be extracted as root; a symlink, hard link or special file, or an++ * absolute or `..` name, could write outside the scratch directory. -> the first offending member.++ */++function tarMemberProblem(listing) {++ for (const line of String(listing || '').split('\n')) {++ if (!line.trim()) continue;++ const m = /^(\S+)\s+\S+\s+\d+\s+\S+\s+\S+\s+(.*)$/.exec(line);++ if (!m) return `unparsable tar listing line: ${line.trim().slice(0, 80)}`;++ const type = m[1][0];++ const name = m[2].replace(/ -> .*$/, '').trim();++ if (type !== '-' && type !== 'd') return `member ${name} is not a regular file or directory (${type})`;++ if (name.startsWith('/') || name.split('/').includes('..')) return `member ${name} is not a safe relative path`;++ }++ return null;++}++// ── inputs ───────────────────────────────────────────────────────────────────/** The backup to restore: --backup <dir>, else the newest usable record in <stateDir>/backups/<id>.jsonl. */async function resolveBackup(exec, inv, svc, dirArg) {let dir = dirArg;let takenAt = null;++ let record = null;const restored = Object.keys(svc.drill.databases).map((name) => svc.databases.find((d) => d.name === name));const pgTargets = restored.filter((d) => d.engine === 'postgresql');if (dir) {@@@ -171,6 +247,7 @@ async function resolveBackup(exec, inv, svc, dirArg) {const last = usable[usable.length - 1];dir = last.dir;takenAt = last.at || null;++ record = last;}const st = await exec.stat(dir);if (!st || !st.isDir) throw new DrillError(`backup directory ${dir} not found`);@@@ -184,7 +261,20 @@ async function resolveBackup(exec, inv, svc, dirArg) {if (db.engine === 'postgresql') files.push({ name, file, production: db, engine: 'postgresql', url: target.url, directUrl: target.directUrl });else files.push({ name, file, production: db.path, envName: target.env, dir: target.dir, ext: path.extname(db.path) || '.db', engine: 'sqlite' });}-- return { dir, takenAt, files };++ // Object directories are restored from the same backup: the archive holds the declared++ // directory's contents, so the drill must not run without it (there would be nothing to verify).++ const objects = [];++ for (const obj of svc.objects) {++ const file = path.join(dir, `${obj.name}.tar.gz`);++ const fst = await exec.stat(file);++ if (!fst || !fst.isFile) throw new DrillError(`backup ${dir} has no copy of the ${obj.name} object directory (${path.basename(file)})`);++ // The uncompressed size recorded at backup time (`du -sb`), so the drill can check it has room++ // to extract. A --backup directory has no record; its compressed size is the best we can do.++ const rec = record && (record.files || []).find((f) => f.file === `${obj.name}.tar.gz`);++ const sourceBytes = rec && Number.isFinite(rec.sourceBytes) ? rec.sourceBytes : fst.size;++ objects.push({ name: obj.name, file, production: obj.path, sourceBytes });++ }++ return { dir, takenAt, files, objects };}/** How production starts the service: ExecStart, WorkingDirectory and non-secret Environment=. */@@@ -287,6 +377,7 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {unit: unitName,pid: null,databases: [],++ objects: [],ready: null,compare: [],counts: [],@@@ -322,6 +413,15 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {for (const name of left.databases) { log(`dropping a scratch database an earlier drill left: ${name}`); await dbengine.dropDatabase(exec, name); }for (const name of left.roles) { log(`dropping a scratch role an earlier drill left: ${name}`); await dbengine.dropRole(exec, name); }}++ // Free space for the object archives too: extracting one writes its whole uncompressed++ // contents under the drill directory. The size `du -sb` measured at backup time travels in the++ // backup record; a --backup directory falls back to the archive's compressed size.++ const objectBytes = bk.objects.reduce((n, o) => n + (o.sourceBytes || 0), 0);++ if (objectBytes) {++ await exec.mkdir(inv.drillDir, { mode: 0o711 });++ const free = (await exec.statfs(inv.drillDir)).free;++ if (free < objectBytes) throw fail('space', `not enough free space under ${inv.drillDir}: ${free} bytes free, need ${objectBytes} (the object directories' contents)`);++ }if (await exec.stat(tmp)) throw fail('prepare', `${tmp} already exists`);await exec.mkdir(inv.drillDir, { mode: 0o711 });// postgres reads the dump copy in <tmp>/pg, so it must be able to traverse <tmp> (0711: no listing;@@@ -385,6 +485,29 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {if (entry.integrity !== 'ok') throw fail('integrity', `${f.name}: integrity_check = ${entry.integrity.slice(0, 300)}`);log(`integrity_check ${f.name}: ok`);}++ // 1b. restore every declared object directory and check every blob's sha256. The archive is++ // root-only, so root extracts it into the drill's own scratch directory — never over the live++ // object directory, whose path reaches the drill instance only through its env override.++ for (const obj of bk.objects) {++ const dest = path.join(tmp, obj.name);++ await exec.mkdir(dest, { owner: svc.runAs, mode: 0o700 });++ // List the archive before extracting as root: only regular files and directories, with++ // relative names, may be written. `--no-same-owner --no-same-permissions` then keeps the++ // archive from restoring production's ownership or modes onto the drill copy.++ const list = await exec.run('tar', ['-tvzf', obj.file], { privileged: true, env: { LC_ALL: 'C' } });++ if (list.code !== 0) throw fail('objects', `listing ${obj.file} failed: ${(list.stderr || '').trim().slice(0, 200)}`);++ if (/Removing leading|Member name contains '\.\.'/.test(list.stderr || '')) throw fail('objects', `${obj.file}: the archive holds an absolute or '..' member`);++ const problem = tarMemberProblem(list.stdout);++ if (problem) throw fail('objects', `${obj.file}: ${problem}`);++ const x = await exec.run('tar', ['-xzf', obj.file, '-C', dest, '--no-same-owner', '--no-same-permissions'], { privileged: true });++ if (x.code !== 0) throw fail('objects', `extracting ${obj.file} failed: ${(x.stderr || '').trim().slice(0, 200)}`);++ const entry = { name: obj.name, source: obj.file, copy: dest, blobs: 0, skipped: 0, bytes: 0, mismatches: 0, integrity: null };++ record.objects.push(entry);++ log(`restoring object directory ${obj.file} → ${dest} (${svc.runAs})`);++ try { await verifyObjectBlobs(exec, dest, entry); } catch (err) { entry.integrity = `error: ${err.message}`; throw fail('objects', `${obj.name}: ${err.message}`); }++ entry.integrity = 'sha256 ok';++ log(`object directory ${obj.name}: ${entry.blobs} blob(s) sha256 ok${entry.skipped ? `, ${entry.skipped} non-blob file(s) skipped` : ''}`);++ }const restoredCounts = [];for (const c of d.counts) {const rdb = svc.databases.find((x) => x.name === c.db);@@@ -589,6 +712,9 @@ function markdownRow(r) {for (const db of r.databases) checks.push(db.engine === 'postgresql'? `\`pg_restore --list\` (${db.name}) = ${db.integrity === 'pg_restore --list ok' ? 'ok' : cell(db.integrity).slice(0, 80)}`: `\`pragma integrity_check\` (${db.name}) = ${db.integrity === 'ok' ? 'ok' : cell(db.integrity).slice(0, 80)}`);++ for (const o of r.objects || []) checks.push(o.integrity === 'sha256 ok'++ ? `object store \`${o.name}\` ${o.blobs} blob(s) sha256 ok`++ : `object store \`${o.name}\`: ${cell(o.integrity).slice(0, 80)}`);if (r.ready) checks.push(r.ready.ok ? `drill instance \`${new URL(r.ready.url).pathname}\` ${r.ready.status} after ${r.ready.seconds}s` : `drill instance not ready (${r.ready.status || 'no answer'})`);const same = r.compare.filter((c) => c.match).map((c) => `\`${c.path}\``);const diff = r.compare.filter((c) => !c.match).map((c) => `\`${c.path}\` differs (${cell(c.detail)})`);@@@ -603,4 +729,4 @@ function markdownRow(r) {return `| ${when} | ${r.service} | \`${r.backup}/\` (\`ovhost drill ${r.service}\`) | ${checks.join('; ')} | ${r.result} |`;}--module.exports = { drill, DrillError, compareBodies };++module.exports = { drill, DrillError, compareBodies, verifyObjectBlobs, tarMemberProblem };@@@ -143,6 +143,26 @@ function normalisePreflight(raw, where) {return out;}++/**++ * `objects`: directories of content-addressed blobs the service owns OUTSIDE its databases, such as++ * OpenVibe.Host's tenant object store (host.example.json: `HOST_STORAGE_DIR`). Each regular file in++ * the directory is named by its own sha256, so `ovhost backup` can archive the tree as one artifact++ * and a drill can check every blob's sha256 after restoring it.++ */++function normaliseObjects(raw, where) {++ if (raw == null) return [];++ if (!Array.isArray(raw)) throw new InventoryError(`${where}.objects must be a list of directories`);++ const seen = new Set();++ return raw.map((o, i) => {++ const at = `${where}.objects[${i}]`;++ if (!o || typeof o !== 'object' || Array.isArray(o)) throw new InventoryError(`${at} must be an object`);++ if (!FILE_NAME_RE.test(o.name || '')) throw new InventoryError(`${at}.name must be a plain name`);++ if (seen.has(o.name)) throw new InventoryError(`${at}.name "${o.name}" is listed twice`);++ seen.add(o.name);++ return { name: o.name, path: abs(o.path, `${at}.path`) };++ });++}++function normaliseService(id, raw, defaults) {if (!ID_RE.test(id)) throw new InventoryError(`service id "${id}" must match ${ID_RE}`);const where = `services.${id}`;@@@ -236,6 +256,9 @@ function normaliseService(id, raw, defaults) {}throw new InventoryError(`${at}.engine must be sqlite or postgresql`);}),++ // Content-addressed object directories alongside the databases (tenant objects, Host's++ // HOST_STORAGE_DIR). Copied as one archive per directory; never opened as a database.++ objects: normaliseObjects(s.objects, where),backupOnChange: s.backupOnChange || [],nginx: s.nginx || null,// The public origin release notifications name (lib/announce.js); default: the manifest's publicOrigin.@@@ -1,10 +1,10 @@'use strict';/**-- * Off-host copies of `ovhost backup --all` runs: every backed-up database is encrypted on the host-- * (lib/backup-crypto.js, AES-256-GCM with a key that stays on the host) and uploaded to S3-compatible-- * object storage (Backblaze B2 in production, the same API Media uses). Nothing is uploaded in the-- * clear except the run manifest, which holds names, sizes and hashes, never data or secrets, and-- * carries an HMAC made with the backup key.++ * Off-host copies of `ovhost backup --all` runs: every backed-up file (a database copy or an object++ * archive) is encrypted on the host (lib/backup-crypto.js, AES-256-GCM with a key that stays on the++ * host) and uploaded to S3-compatible object storage (Backblaze B2 in production, the same API Media++ * uses). Nothing is uploaded in the clear except the run manifest, which holds names, sizes and hashes,++ * never data or secrets, and carries an HMAC made with the backup key.** Configuration comes from a root-only env file (default /etc/openvibe/backup.env):*@@@ -252,7 +252,7 @@ async function push(ctx, summary, { cfg, client, partSize = PART_SIZE, chunkSizeawait pipeline(exec.readStream(f.dest), plain, bcrypto.createEncryptStream(master, chunkSize ? { chunkSize } : {}), cipher, async (source) => { up = await uploadIterable(client, cfg.bucket, key, source, { partSize }); });const p = plain.result();const c = cipher.result();-- const entry = { service: svc.service, name: f.name, file: name, object: key, source: f.source || null, plainBytes: p.bytes, plainSha256: p.sha256, cipherBytes: c.bytes, cipherSha256: c.sha256, parts: up.parts, seconds: Math.round((exec.now() - t0) / 1000) };++ const entry = { service: svc.service, name: f.name, kind: f.kind || 'database', file: name, object: key, source: f.source || null, plainBytes: p.bytes, plainSha256: p.sha256, cipherBytes: c.bytes, cipherSha256: c.sha256, parts: up.parts, seconds: Math.round((exec.now() - t0) / 1000) };result.uploaded.push(entry);(manifest.services[svc.service] = manifest.services[svc.service] || { files: [] }).files.push(entry);log(`uploaded ${f.dest} → s3://${cfg.bucket}/${key} (${c.bytes} bytes encrypted)`);@@@ -322,7 +322,7 @@ async function readManifest(client, cfg, master, host, run) {return m;}--/** Directories restore-download must never write into: anything holding a declared database. */++/** Directories restore-download must never write into: anything holding a declared database or object directory. */function unsafeOut(inv, out) {for (const svc of Object.values(inv.services)) {for (const db of svc.databases) {@@@ -331,6 +331,11 @@ function unsafeOut(inv, out) {const dir = path.dirname(db.path);if (out === dir || db.path.startsWith(`${out}/`) || out.startsWith(`${dir}/`)) return `${out} holds or sits inside ${svc.id}'s database directory ${dir}`;}++ // An object directory is itself a directory the live service writes: never restore over it.++ for (const obj of svc.objects || []) {++ const dir = obj.path;++ if (out === dir || dir.startsWith(`${out}/`) || out.startsWith(`${dir}/`)) return `${out} holds or sits inside ${svc.id}'s object directory ${dir}`;++ }if (out === svc.repo || out.startsWith(`${svc.repo}/`) || svc.repo.startsWith(`${out}/`)) return `${out} overlaps ${svc.id}'s checkout ${svc.repo}`;}for (const d of [inv.backupDir, inv.stateDir]) if (out === d || d.startsWith(`${out}/`) || out.startsWith(`${d}/`)) return `${out} overlaps ${d}`;@@@ -390,18 +395,25 @@ async function restoreDownload(ctx, id, runArg, { cfg, client, out = null, host:}// A PostgreSQL artifact is a pg_dump archive, checked by parsing it (pg_restore --list, which// needs no database and is run as the caller — the copy is root-only 0600, and postgres could-- // not read it); a SQLite copy is quick_checked through the worker.++ // not read it); a SQLite copy is quick_checked through the worker. An object-directory archive++ // is neither: its own sha256 has already matched the manifest, and a drill is what extracts it++ // and checks every blob.let check;const isDump = path.extname(f.file) === '.dump';-- try {-- if (isDump) { await dbengine.verifyDump(exec, dest, { as: null }); check = 'pg_restore --list ok'; }-- else {-- const rows = await exec.sqlite(dest, 'PRAGMA quick_check');-- check = rows && rows[0] ? String(Object.values(rows[0])[0]) : 'no answer';-- }-- } catch (err) { check = `error: ${err.message}`; }-- record.files.push({ name: f.name, file: dest, source: f.source, bytes: p.bytes, sha256: p.sha256, quickCheck: check });-- if (isDump ? check.startsWith('error') : check !== 'ok') throw new OffsiteError(`${dest}: ${check}`);++ const isObjects = f.kind === 'objects' || f.file.endsWith('.tar.gz');++ if (isObjects) {++ check = 'object archive (verify with a drill)';++ } else {++ try {++ if (isDump) { await dbengine.verifyDump(exec, dest, { as: null }); check = 'pg_restore --list ok'; }++ else {++ const rows = await exec.sqlite(dest, 'PRAGMA quick_check');++ check = rows && rows[0] ? String(Object.values(rows[0])[0]) : 'no answer';++ }++ } catch (err) { check = `error: ${err.message}`; }++ }++ record.files.push({ name: f.name, kind: isObjects ? 'objects' : 'database', file: dest, source: f.source, bytes: p.bytes, sha256: p.sha256, quickCheck: isObjects ? null : check });++ if (!isObjects && (isDump ? check.startsWith('error') : check !== 'ok')) throw new OffsiteError(`${dest}: ${check}`);}return record;}@@@ -15,10 +15,12 @@ const retention = require('../lib/retention');const offsite = require('../lib/offsite');const { normalise } = require('../lib/inventory');const { mockS3 } = require('./s3-mock');++const { readTarGz } = require('../lib/tar');const DAY = 86400000;const KEY_HEX = crypto.createHash('sha256').update('test backup key').digest('hex');const S3_SECRET = `${SECRET}-s3`;++const TENANT_BLOB = 'tenant object bytes\n';/** The standard fake host plus AI (two databases) and OpenRe (not deployed), a backup env and key. */function backupHost({ envMode = 0o600, keyMode = 0o600, envOwner = 'root', env = null } = {}) {@@@ -75,6 +77,25 @@ function noSecrets(text, what) {for (const s of [S3_SECRET, SECRET, KEY_HEX]) assert.ok(!String(text).includes(s), `${what} leaked a secret value`);}++/** backupHost plus a service "host" on PostgreSQL with a declared content-addressed object directory. */++function objectsHost() {++ const host = backupHost();++ const doc = JSON.parse(host.read('/etc/openvibe/host.json'));++ doc.services.host = {++ repo: '/opt/openvibe.host',++ units: [],++ databases: [{ name: 'host', engine: 'postgresql', database: 'ov_host' }],++ objects: [{ name: 'objects', path: '/var/lib/openvibe-host-api/objects' }],++ };++ host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });++ host.inv = normalise(doc);++ host.pgDatabases.add('ov_host');++ host.pgBackups = [{ type: 'diff', timestamp: { stop: Math.floor(host.exec.now() / 1000) - 3600 } }];++ host.tenantSha = crypto.createHash('sha256').update(TENANT_BLOB).digest('hex');++ host.put(`/var/lib/openvibe-host-api/objects/projects/prj_1/${host.tenantSha.slice(0, 2)}/${host.tenantSha}`, TENANT_BLOB, { owner: 'ubuntu', mode: 0o644 });++ return host;++}++async function decrypt(key, buf) {const chunks = [];await pipeline(Readable.from([buf]), bcrypto.createDecryptStream(key), async (src) => { for await (const c of src) chunks.push(c); });@@@ -355,6 +376,83 @@ runTests([assert.strictEqual(s3.objects.size, 0);}),++ test('--offsite: a declared object directory is archived, encrypted, uploaded and recorded in the manifest', async () => {++ const host = objectsHost();++ const r = await host.cli('backup', '--all', '--offsite');++ assert.strictEqual(r.code, 0, r.out);++ // The archive is an ordinary root-only copy in the local backup...++ const local = host.files.get('/var/backups/openvibe/host/20260922-120000/objects.tar.gz');++ assert.deepStrictEqual([local.owner, local.mode], ['root', 0o600]);++ // ...and an encrypted object off-host, under the same rules as every other file.++ const base = 'openvibe-backups/fake-host/20260922-120000/';++ const key = `${base}host/objects.tar.gz.ovbk`;++ assert.ok(host.s3.objects.has(key), [...host.s3.objects.keys()].join('\n'));++ const enc = host.s3.objects.get(key);++ assert.ok(!enc.includes('tenant object'), 'the archive is not encrypted');++ const plain = await decrypt(Buffer.from(KEY_HEX, 'hex'), enc);++ const entries = readTarGz(plain);++ assert.deepStrictEqual(entries.map((e) => e.name), [`projects/prj_1/${host.tenantSha.slice(0, 2)}/${host.tenantSha}`]);++ assert.strictEqual(entries[0].content, TENANT_BLOB);++ const m = JSON.parse(host.s3.objects.get(`${base}manifest.json`));++ const f = m.services.host.files.find((x) => x.file === 'objects.tar.gz');++ assert.deepStrictEqual([f.kind, f.name], ['objects', 'objects']);++ assert.strictEqual(f.plainSha256, crypto.createHash('sha256').update(plain).digest('hex'));++ noSecrets(host.s3.objects.get(`${base}manifest.json`), 'the manifest');++ noSecrets(JSON.stringify(host.s3.sent), 'S3 requests');++ noSecrets(r.out, 'CLI output');++ }),++++ test('an object archive is written only after root has taken the staging directory over', async () => {++ const host = objectsHost();++ const r = await host.cli('backup', 'host');++ assert.strictEqual(r.code, 0, r.out);++ const tarCall = host.calls.find((c) => c.cmd === 'tar' && c.args.some((a) => String(a).endsWith('objects.tar.gz')));++ assert.ok(tarCall, 'the object directory was tarred');++ assert.strictEqual(tarCall.privileged, true);++ const stage = path.dirname(tarCall.args[1]);++ const lockAt = host.calls.findIndex((c) => c.cmd === 'chown' && c.args[c.args.length - 1] === stage && c.args.includes('root:root'));++ const chmodAt = host.calls.findIndex((c) => c.cmd === 'chmod' && c.args[c.args.length - 1] === stage && c.args.includes('0700'));++ const tarAt = host.calls.indexOf(tarCall);++ assert.ok(lockAt >= 0 && chmodAt >= 0, 'root locked the staging directory down');++ assert.ok(lockAt < tarAt && chmodAt < tarAt, 'root owned the stage before tar opened the archive');++ }),++++ test('a tar warning for transient scratch under the store is accepted, and the scratch excluded', async () => {++ const host = objectsHost();++ host.put('/var/lib/openvibe-host-api/objects/tmp/aaaaaaaaaaaa.part', 'half-written upload', { owner: 'ubuntu', mode: 0o644 });++ host.onTar = (args) => (String(args[0]).includes('c') ? { code: 1, stderr: 'tar: ./tmp/aaaaaaaaaaaa.part: file changed as we read it\n' } : undefined);++ const r = await host.cli('backup', 'host', '--json');++ host.onTar = null;++ assert.strictEqual(r.code, 0, r.out);++ const rec = JSON.parse(r.out);++ const f = rec.files.find((x) => x.kind === 'objects');++ assert.ok(f && !f.error, JSON.stringify(f));++ assert.ok(Number.isFinite(f.sourceBytes) && f.sourceBytes > 0, 'the source size was recorded for the drill');++ const tarCall = host.calls.find((c) => c.cmd === 'tar' && c.args.some((a) => String(a).endsWith('.tar.gz')));++ assert.ok(tarCall.args.includes('--exclude=./tmp'), tarCall.args.join(' '));++ const names = readTarGz(host.files.get(`${rec.dir}/objects.tar.gz`).content).map((e) => e.name);++ assert.ok(!names.some((n) => n.startsWith('tmp/')), `scratch was archived: ${names.join(', ')}`);++ assert.ok(names.length >= 1, 'the tenant blob is still archived');++ }),++++ test('a genuine tar failure still fails the backup', async () => {++ const host = objectsHost();++ host.onTar = (args) => (String(args[0]).includes('c') ? { code: 2, stderr: 'tar: ./projects: Cannot open: Permission denied\n' } : undefined);++ const r = await host.cli('backup', 'host');++ host.onTar = null;++ assert.strictEqual(r.code, 2, r.out);++ assert.match(r.out, /Permission denied/);++ }),++++ test('an object directory is not archived when the staging filesystem has no room', async () => {++ const host = objectsHost();++ host.statfsFree = 1024;++ const r = await host.cli('backup', 'host');++ assert.strictEqual(r.code, 2, r.out);++ assert.match(r.out, /not enough free space under \/var\/backups\/openvibe\.staging/);++ assert.ok(!host.calls.some((c) => c.cmd === 'tar'), 'nothing was tarred');++ }),++test('off-host retention: runs older than 30 days are deleted, the newest 7 always stay, and nothing is pruned after a failed upload', async () => {const host = backupHost();const put = (run) => {@@@ -487,6 +585,22 @@ runTests([assert.match(refused.out, /pg_restore --list .* failed|text format dump/);}),++ test('restore-download: an object archive is written without a database check and never over the live object directory', async () => {++ const host = objectsHost();++ await host.cli('backup', '--all', '--offsite');++ const near = await host.cli('restore-download', 'host', 'latest', '--out', '/var/lib/openvibe-host-api/objects');++ assert.strictEqual(near.code, 1, near.out);++ assert.match(near.out, /refusing --out: .*object directory/);++ const r = await host.cli('restore-download', 'host', 'latest', '--out', '/root/r-host', '--json');++ assert.strictEqual(r.code, 0, r.out);++ const rec = JSON.parse(r.out);++ const f = rec.files.find((x) => x.file.endsWith('objects.tar.gz'));++ assert.deepStrictEqual([f.kind, f.quickCheck], ['objects', null], 'an object archive is not quick_checked as a database');++ assert.ok(host.files.has('/root/r-host/objects.tar.gz'));++ assert.ok(!host.sqliteCalls.some((c) => c.db === '/root/r-host/objects.tar.gz'));++ assert.ok(rec.files.some((x) => x.file.endsWith('host.dump') && x.kind === 'database'), 'the database is still checked');++ }),++// ── configuration ──────────────────────────────────────────────────────────test('config: root-only env file and key; missing names listed by name; not root refused; no value ever printed', async () => {let host = backupHost({ envMode: 0o644 });@@@ -6,16 +6,24 @@* the drill log is written.*/const assert = require('assert');++const crypto = require('crypto');const fs = require('fs');const path = require('path');const { scenario, test, runTests, SECRET } = require('./helpers');const { normalise } = require('../lib/inventory');--const { compareBodies } = require('../lib/commands/drill');++const { compareBodies, verifyObjectBlobs } = require('../lib/commands/drill');const DRILL_DIR = '/var/lib/openvibe-drills';const PROD_DB = '/var/lib/openvibe-tools/tools.db';++const OBJECTS_DIR = '/var/lib/openvibe-host-api/objects';const PASTES = '[{"id":"pst_1","title":"hello"},{"id":"pst_2","title":"world"}]';++/** A content-addressed blob under the Host object-store layout: <root>/projects/<prj>/<aa>/<sha256>. */++function blob(root, data) {++ const sha = crypto.createHash('sha256').update(data).digest('hex');++ return { sha, path: `${root}/projects/prj_1/${sha.slice(0, 2)}/${sha}` };++}++function toolsEntry(drillOverrides = {}) {return {repo: '/opt/openvibe.tools',@@@ -40,10 +48,12 @@ function toolsEntry(drillOverrides = {}) {* The standard fake host plus Tools in production, with one `ovhost backup tools` taken.* The drill instance's behaviour is controlled through `host.drill`.*/--async function drillScenario({ drill: drillOverrides, backup = true } = {}) {++async function drillScenario({ drill: drillOverrides, backup = true, objects = null, blobs = [] } = {}) {const host = scenario();const doc = JSON.parse(host.read('/etc/openvibe/host.json'));-- doc.services.tools = toolsEntry(drillOverrides);++ const tools = toolsEntry(drillOverrides);++ if (objects) tools.objects = objects;++ doc.services.tools = tools;host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });const repo = host.createRepo('/opt/openvibe.tools', { owner: 'ubuntu' });@@@ -100,6 +110,8 @@ async function drillScenario({ drill: drillOverrides, backup = true } = {}) {return undefined;};++ for (const b of blobs) host.put(b.path, b.content, { owner: 'ubuntu', mode: 0o644 });++if (backup) {const b = await host.cli('backup', 'tools', '--json');assert.strictEqual(b.code, 0, b.out);@@@ -545,6 +557,137 @@ runTests([normalise({ services: { apps: multi } }); // valid inventory; the drill itself refuses (below)}),++ test('objects: the declared object directory is archived, restored into scratch, and every blob sha256 checked', async () => {++ const a = 'tenant page a\n';++ const b = 'tenant asset b\n';++ const pa = blob(OBJECTS_DIR, a);++ const pb = blob(OBJECTS_DIR, b);++ const host = await drillScenario({++ objects: [{ name: 'objects', path: OBJECTS_DIR }],++ blobs: [{ path: pa.path, content: a }, { path: pb.path, content: b }],++ drill: {++ env: { PORT: '{port}', HOST: '127.0.0.1', DISCORD_RELAY_ENABLED: '0', EVENTS_URL: '', UPLOAD_DIR: '{tmp}/data/uploads', GREETING: 'two words', HOST_STORAGE_DIR: '{tmp}/objects' },++ dirs: ['{tmp}/data', '{tmp}/objects'],++ },++ });++ const archive = `${host.backupDir}/objects.tar.gz`;++ assert.deepStrictEqual([host.files.get(archive).owner, host.files.get(archive).mode], ['root', 0o600], 'the archive is an ordinary root-only copy');++ const r = await host.cli('drill', 'tools', '--keep');++ assert.strictEqual(r.code, 0, r.out);++ const rec = lastLog(host);++ assert.strictEqual(rec.kept, true);++ assert.deepStrictEqual(rec.objects.map((o) => [o.name, o.blobs, o.skipped, o.integrity]), [['objects', 2, 0, 'sha256 ok']]);++ const tmp = rec.dir;++ // The archive is listed first, then extracted without restoring production's ownership or modes.++ assert.ok(host.calls.some((c) => c.cmd === 'tar' && c.args[0] === '-tvzf' && c.args[1] === archive), 'listed before extraction');++ const extract = host.calls.find((c) => c.cmd === 'tar' && c.args[0] === '-xzf');++ assert.deepStrictEqual(extract.args, ['-xzf', archive, '-C', `${tmp}/objects`, '--no-same-owner', '--no-same-permissions']);++ assert.strictEqual(extract.privileged, true);++ // Restored into the drill's own scratch directory, never the live object directory.++ assert.strictEqual(rec.objects[0].copy, `${tmp}/objects`);++ assert.ok(rec.objects[0].copy.startsWith(`${tmp}/`));++ assert.strictEqual(host.read(`${tmp}/objects/projects/prj_1/${pa.sha.slice(0, 2)}/${pa.sha}`), a);++ assert.strictEqual(host.read(`${tmp}/objects/projects/prj_1/${pb.sha.slice(0, 2)}/${pb.sha}`), b);++ // The instance is pointed at the restored directory, not production's.++ assert.ok(host.read(`${tmp}/drill.env`).split('\n').includes(`HOST_STORAGE_DIR=${tmp}/objects`));++ assert.match(rec.markdown, /object store `objects` 2 blob\(s\) sha256 ok/);++ // The live objects were never written or moved.++ assert.strictEqual(host.read(pa.path), a);++ assert.strictEqual(host.read(pb.path), b);++ }),++++ test('objects: a blob whose bytes do not match the sha256 in its name fails the drill before anything starts', async () => {++ const good = 'good blob\n';++ const p = blob(OBJECTS_DIR, good);++ const wrongName = 'a'.repeat(64);++ const host = await drillScenario({++ objects: [{ name: 'objects', path: OBJECTS_DIR }],++ blobs: [{ path: p.path, content: good }, { path: `${OBJECTS_DIR}/projects/prj_1/aa/${wrongName}`, content: 'not its name\n' }],++ });++ const before = fsSnapshot(host);++ const r = await host.cli('drill', 'tools');++ assert.strictEqual(r.code, 2, r.out);++ const rec = lastLog(host);++ assert.strictEqual(rec.failure.stage, 'objects');++ assert.match(rec.failure.message, /blob\(s\) do not match their name/);++ assert.match(rec.failure.message, /aa\/a{64}/);++ assert.strictEqual(host.systemdRuns.length, 0, 'nothing is started');++ assert.ok(!host.files.has(rec.dir));++ assert.strictEqual(host.read(p.path), good, 'the live object is untouched');++ for (const changed of changedPaths(before, fsSnapshot(host))) assert.ok(allowedWrite(changed, rec.dir), `unexpected write: ${changed}`);++ }),++++ test('objects: a service declaring an object directory refuses to drill from a backup without its archive', async () => {++ // The directory does not exist, so the backup records the object directory as skipped.++ const host = await drillScenario({ objects: [{ name: 'objects', path: OBJECTS_DIR }], blobs: [] });++ const r = await host.cli('drill', 'tools');++ assert.strictEqual(r.code, 1);++ assert.match(r.out, /has no copy of the objects object directory \(objects\.tar\.gz\)/);++ assert.strictEqual(host.systemdRuns.length, 0);++ }),++++ test('objects: a symlink or special-file member is refused before root extracts the archive', async () => {++ for (const member of [{ type: 'l', name: './evil', target: '/' }, { type: 'p', name: './fifo' }]) {++ const a = 'tenant page a\n';++ const pa = blob(OBJECTS_DIR, a);++ const host = await drillScenario({ objects: [{ name: 'objects', path: OBJECTS_DIR }], blobs: [{ path: pa.path, content: a }] });++ host.tarMembers = [member];++ const r = await host.cli('drill', 'tools');++ assert.strictEqual(r.code, 2, `${member.type}: ${r.out}`);++ const rec = lastLog(host);++ assert.strictEqual(rec.failure.stage, 'objects');++ assert.match(rec.failure.message, /not a regular file or directory/);++ assert.ok(!host.calls.some((c) => c.cmd === 'tar' && c.args[0] === '-xzf'), `${member.type}: never extracted`);++ assert.ok(!host.files.has(rec.dir), 'cleaned up');++ }++ }),++++ test('objects: a symlinked directory already in the restored tree is refused, not walked as root', async () => {++ const host = scenario();++ const dir = '/var/lib/openvibe-drills/tools-20260922-120100/objects';++ const good = 'a good blob\n';++ const sha = crypto.createHash('sha256').update(good).digest('hex');++ host.put(`${dir}/projects/prj_1/${sha.slice(0, 2)}/${sha}`, good, { owner: 'ubuntu' });++ // A file whose name IS a sha256 but whose bytes are not, behind a symlinked directory: if the++ // walk followed the link it would hash this and report a mismatch.++ const outsideSha = 'b'.repeat(64);++ host.put(`/var/lib/openvibe-secret/${outsideSha}`, 'not its name\n', { owner: 'root', mode: 0o600 });++ await host.exec.symlink('/var/lib/openvibe-secret', `${dir}/evil`);++ const entry = { blobs: 0, skipped: 0, bytes: 0, mismatches: 0 };++ await assert.rejects(++ verifyObjectBlobs(host.exec, dir, entry),++ (err) => /evil \(symlink or special file\)/.test(err.message) && !err.message.includes(outsideSha) && !err.message.includes('/var/lib/openvibe-secret'),++ );++ assert.strictEqual(entry.blobs, 1, 'the real blob was still checked');++ assert.strictEqual(entry.mismatches, 1);++ }),++++ test('objects: a drill with no room to extract the archive fails before root restores it', async () => {++ const a = 'tenant page a\n';++ const pa = blob(OBJECTS_DIR, a);++ const host = await drillScenario({ objects: [{ name: 'objects', path: OBJECTS_DIR }], blobs: [{ path: pa.path, content: a }] });++ const callsBefore = host.calls.length;++ host.statfsFree = 1024;++ const r = await host.cli('drill', 'tools');++ assert.strictEqual(r.code, 2, r.out);++ const rec = lastLog(host);++ assert.strictEqual(rec.failure.stage, 'space');++ assert.match(rec.failure.message, /not enough free space under \/var\/lib\/openvibe-drills/);++ assert.ok(!host.calls.slice(callsBefore).some((c) => c.cmd === 'tar'), 'nothing was tarred or extracted');++ assert.ok(!host.files.has(rec.dir));++ }),++++ test('inventory: object directories are validated', () => {++ const base = { owner: 'ubuntu', repo: '/opt/openvibe.host', units: [], databases: [{ name: 'host', engine: 'postgresql', database: 'ov_host' }] };++ const svc = (objects) => normalise({ services: { h: { ...base, objects } } });++ assert.deepStrictEqual(svc([{ name: 'objects', path: '/var/lib/openvibe-host-api/objects' }]).services.h.objects, [{ name: 'objects', path: '/var/lib/openvibe-host-api/objects' }]);++ assert.deepStrictEqual(svc(undefined).services.h.objects, []);++ assert.throws(() => svc([{ name: 'a/b', path: '/x' }]), /name must be a plain name/);++ assert.throws(() => svc([{ name: 'objects', path: 'relative' }]), /absolute path/);++ assert.throws(() => svc([{ name: 'objects', path: '/x' }, { name: 'objects', path: '/y' }]), /listed twice/);++ assert.throws(() => svc('nope'), /must be a list of directories/);++ }),++test('several units and neither drill.unit nor drill.command: refused', async () => {const host = await appsScenario({ drill: { unit: null } });const r = await host.cli('drill', 'apps');@@@ -13,6 +13,7 @@const crypto = require('crypto');const path = require('path');const { Readable, Writable } = require('stream');++const { createTarGz, readTarGz } = require('../lib/tar');function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', start = Date.parse('2026-09-22T12:00:00Z') } = {}) {const files = new Map(); // path -> { type: 'file'|'dir'|'symlink', content, mode, owner, target }@@@ -53,6 +54,8 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', ststatfsFree: 100 * 1024 ** 3,statfsSize: 500 * 1024 ** 3,reads: [],++ onTar: null, // (args, opts, real) -> undefined | { code, stdout, stderr } ; override a tar result++ tarMembers: [], // [{ type, name, target }] extra (non-)members a `tar -tvzf` listing reportscertbot: null, // (args, opts) -> { code, stdout, stderr } | Error ; default: nothing to docertbotCalls: [], // { args, as, privileged }journalctl: null, // (args, opts) -> { code, stdout, stderr } ; default: no entries@@@ -365,6 +368,66 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', streturn { code: 0, stdout: 'Done', stderr: '' };}++ // ── tar (backup/drill of a content-addressed object directory) ──++ // The forms the backup and drill use: `tar -czf <dest> -C <dir> .`, `tar -tvzf <archive>` (the++ // drill's member pre-check) and `tar -xzf <archive> -C <dir>`. The archive is a real gzipped tar++ // (lib/tar.js), so it survives the off-host encrypt/decrypt pipeline byte for byte.++ function tarCmd(args, opts) {++ const result = tarCmdReal(args, opts);++ // A test can override the result (to simulate a warning or a failure) after the real archive++ // was made, so --exclude and the archive's contents stay realistic.++ if (host.onTar) { const r = host.onTar(args, opts, result); if (r) return { stdout: '', stderr: '', ...r }; }++ return result;++ }++ function tarCmdReal(args, opts) {++ const mode = String(args[0] || '');++ const archive = args[1];++ const ci = args.indexOf('-C');++ const dir = ci >= 0 ? args[ci + 1] : '.';++ // `--exclude=PAT` (used for Host's transient <root>/tmp scratch): skip a member whose relative++ // name is the pattern or sits under it.++ const excludes = args.filter((a) => a.startsWith('--exclude=')).map((a) => a.slice(10).replace(/^\.\//, ''));++ const excluded = (name) => excludes.some((ex) => name === ex || name.startsWith(`${ex}/`));++ if (mode.includes('c')) {++ const target = args[args.length - 1];++ const base = resolveLinks(dir);++ const root = resolveLinks(path.join(dir, target));++ const entries = [];++ for (const [k, v] of files) {++ if (v.type !== 'file') continue;++ if (k !== root && !k.startsWith(`${root}/`)) continue;++ const name = path.relative(base, k);++ if (excluded(name)) continue;++ entries.push({ name, content: v.content });++ }++ put(archive, createTarGz(entries, clock), { owner: opts.as || user, mode: 0o600 });++ return { code: 0, stdout: '', stderr: '' };++ }++ if (mode.includes('t')) {++ const e = get(archive);++ if (!e || e.type !== 'file') return { code: 1, stdout: '', stderr: `tar: ${archive}: No such file or directory` };++ let parsed;++ try { parsed = readTarGz(Buffer.isBuffer(e.content) ? e.content : Buffer.from(String(e.content))); } catch (err) { return { code: 1, stdout: '', stderr: `tar: ${err.message}` }; }++ const lines = parsed.map((f) => `-rw-r--r-- ubuntu/ubuntu ${Buffer.byteLength(f.content)} 2026-09-22 12:00 ${f.name}`);++ for (const m of host.tarMembers) lines.push(`${m.type}rw-r--r-- root/root 0 2026-09-22 12:00 ${m.name}${m.target ? ` -> ${m.target}` : ''}`);++ return { code: 0, stdout: lines.length ? `${lines.join('\n')}\n` : '', stderr: '' };++ }++ if (mode.includes('x')) {++ const e = get(archive);++ if (!e || e.type !== 'file') return { code: 1, stdout: '', stderr: `tar: ${archive}: No such file or directory` };++ const destdir = resolveLinks(dir);++ let parsed;++ try { parsed = readTarGz(Buffer.isBuffer(e.content) ? e.content : Buffer.from(String(e.content))); } catch (err) { return { code: 1, stdout: '', stderr: `tar: ${err.message}` }; }++ for (const f of parsed) {++ const name = f.name.replace(/^\.\//, '');++ if (!name || name.split('/').includes('..')) continue;++ put(path.join(destdir, name), f.content, { owner: opts.as || 'root', mode: 0o644 });++ }++ return { code: 0, stdout: '', stderr: '' };++ }++ return { code: 1, stdout: '', stderr: `fake tar: unsupported ${args.join(' ')}` };++ }++// ── systemd ──host.addUnit = (unit, state = {}) => {units.set(unit, { load: 'loaded', active: 'active', sub: 'running', mainPid: 1000 + units.size, fragmentPath: `/etc/systemd/system/${unit}`, dropIns: [], partOf: [], restarts: 0, runningSha: null, ...state });@@@ -499,6 +562,17 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', stconst r = host.onNode ? host.onNode(args, opts) : undefined;return r ? { stdout: '', stderr: '', ...r } : { code: 0, stdout: '', stderr: '' };}++ case 'du': {++ const target = args[args.length - 1];++ const root = resolveLinks(target);++ if (!files.has(root)) return { code: 1, stdout: '', stderr: `du: cannot access '${target}': No such file or directory` };++ let total = 0;++ for (const [k, v] of files) {++ if (k !== root && !k.startsWith(`${root}/`)) continue;++ total += v.type === 'file' ? Buffer.byteLength(Buffer.isBuffer(v.content) ? v.content : String(v.content)) : 4096;++ }++ return { code: 0, stdout: `${total}\t${target}\n`, stderr: '' };++ }case 'certbot': {host.certbotCalls.push({ args, as: opts.as || null, privileged: !!opts.privileged });const r = host.certbot ? host.certbot(args, opts) : undefined;@@@ -572,6 +646,7 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', streturn { code: 0, stdout: `${JSON.stringify(stanzas)}\n`, stderr: '' };}case 'systemd-run': return systemdRunCmd(args);++ case 'tar': return tarCmd(args, opts);case 'cp': {const src = args[args.length - 2];const dest = args[args.length - 1];
@@ -404,11 +404,11 @@ The service runs on the host (loopback, since 2026-09-23) and the pending tenant
@@ -404,11 +404,11 @@ The service runs on the host (loopback, since 2026-09-23) and the pending tenant
### Not done yet (Stage B)
### Not done yet (Stage B)
- Uploads are held in memory while they are validated (bounded by `HOST_MAX_UPLOAD_BYTES` and `HOST_MAX_UNPACKED_BYTES`), so the service needs that much headroom per concurrent upload.
- Uploads are held in memory while they are validated (bounded by `HOST_MAX_UPLOAD_BYTES` and `HOST_MAX_UNPACKED_BYTES`), so the service needs that much headroom per concurrent upload.
-- Objects live on the service's local disk, not in OpenVibe.Media; there is no replication and no backup beyond `ovhost backup host`. The database (`ov_host` on the host's data role, ADR-035; schema in [migrations/](migrations/)) is backed up with the others by pgBackRest.
+- Objects live on the service's local disk, not in OpenVibe.Media; there is no replication. `ovhost backup host` archives `/var/lib/openvibe-host-api/objects` (the inventory's `objects` entry) alongside `ov_host`, uploads the encrypted archive off-host, and a restore drill extracts it and checks every blob's sha256. The database (`ov_host` on the host's data role, ADR-035; schema in [migrations/](migrations/)) is backed up with the others by pgBackRest.
- Certificates for custom domains are issued and installed by `ovhost certs renew --install` (step 8) on the `openvibe-certs.timer`; certbot stays outside the Host service and no key file is read.
- Certificates for custom domains are issued and installed by `ovhost certs renew --install` (step 8) on the `openvibe-certs.timer`; certbot stays outside the Host service and no key file is read.
- Projects are created in Host. When Network has projects (ADR-014), Host should accept only Network project ids and read membership from Network.
- Projects are created in Host. When Network has projects (ADR-014), Host should accept only Network project ids and read membership from Network.
- `openvibe.host` is not on the Public Suffix List, so a tenant page can still set `Domain=openvibe.host` cookies (a cookie bomb breaks the dashboard and other tenant sites for that visitor). The protections above do not depend on it. [docs/threat-review.md §5](docs/threat-review.md#5-the-public-suffix-list-question) records the decision (launch without it) and the recommended follow-up (dashboard off the tenant zone, then list the zone).
- `openvibe.host` is not on the Public Suffix List, so a tenant page can still set `Domain=openvibe.host` cookies (a cookie bomb breaks the dashboard and other tenant sites for that visitor). The protections above do not depend on it. [docs/threat-review.md §5](docs/threat-review.md#5-the-public-suffix-list-question) records the decision (launch without it) and the recommended follow-up (dashboard off the tenant zone, then list the zone).
-- Tenant objects are not in `ovhost backup` (only `host.db`).
+- Tenant objects are in `ovhost backup`: the inventory declares them as `objects` (`/var/lib/openvibe-host-api/objects`), the backup archives the tree, the off-host copy encrypts it, and a drill extracts it and checks every blob's sha256.
- Git deploys are Phase 1 only (see [Git deploys](#git-deploys-phase-1)): the project's own CI builds and posts the output. Host never clones, fetches, holds a repository credential or builds; in-Host builds belong to Run (plan T14). There is no webhook: a push reaches Host only through the CI's call.
- Git deploys are Phase 1 only (see [Git deploys](#git-deploys-phase-1)): the project's own CI builds and posts the output. Host never clones, fetches, holds a repository credential or builds; in-Host builds belong to Run (plan T14). There is no webhook: a push reaches Host only through the CI's call.
- Tenant sites publish their own sitemap/robots/feed; when a deploy ships no `sitemap.xml` or `robots.txt`, Host generates one (the manifest's HTML pages on the host the request came in on, so a verified custom domain gets its own `sitemap.xml`; sandbox projects get `Disallow: /` and are `noindex`). A tenant's own file at either path always wins. The dashboard host serves `/robots.txt` and `/sitemap.xml` (front page and legal pages).
- Tenant sites publish their own sitemap/robots/feed; when a deploy ships no `sitemap.xml` or `robots.txt`, Host generates one (the manifest's HTML pages on the host the request came in on, so a verified custom domain gets its own `sitemap.xml`; sandbox projects get `Disallow: /` and are `noindex`). A tenant's own file at either path always wins. The dashboard host serves `/robots.txt` and `/sitemap.xml` (front page and legal pages).
- The Codes portal (Wave 20) does not exist, so there is no public developer onboarding for Host yet.
- The Codes portal (Wave 20) does not exist, so there is no public developer onboarding for Host yet.
@@ -14,7 +14,7 @@ Restore drills (a second instance started from a backup and compared with produc
@@ -14,7 +14,7 @@ Restore drills (a second instance started from a backup and compared with produc
## What is backed up
## What is backed up
`ovhost backup --all` backs up every service in the inventory (`/etc/openvibe/host.json`) that has
`ovhost backup --all` backs up every service in the inventory (`/etc/openvibe/host.json`) that has
-a `databases` list. `databases[]` items are one of two **engines** ([db-inventory.md](db-inventory.md)):
+a `databases` or `objects` list. `databases[]` items are one of two **engines** ([db-inventory.md](db-inventory.md)):
SQLite, a file the service user's worker copies; and PostgreSQL, a database in the host's cluster
SQLite, a file the service user's worker copies; and PostgreSQL, a database in the host's cluster
(ADR-035) that ovhost verifies through pgBackRest and periodically dumps. With `host.example.json`:
(ADR-035) that ovhost verifies through pgBackRest and periodically dumps. With `host.example.json`:
@@ -34,6 +34,13 @@ services](#postgresql-services) covers what that means.
@@ -34,6 +34,13 @@ services](#postgresql-services) covers what that means.
A database that does not exist yet (a service that is not deployed) is reported as `skipped` and
A database that does not exist yet (a service that is not deployed) is reported as `skipped` and
does not fail the run. Adding a database to the inventory is all it takes to include it.
does not fail the run. Adding a database to the inventory is all it takes to include it.
+**Object directories.** A service may also declare `objects`: a list of `{ name, path }` directories
+whose files are content-addressed, each named by its own sha256. OpenVibe.Host's tenant object store
+(`HOST_STORAGE_DIR=/var/lib/openvibe-host-api/objects`) is the first. `ovhost backup` archives each
+directory as `<name>.tar.gz` (root:root 0600, like every other copy); the off-host code encrypts and
+uploads it with the rest, and a [drill](restore-drills.md) extracts it into its own scratch directory
+and checks every blob's sha256.
+
## One run
## One run
For each service, one after another:
For each service, one after another:
@@ -448,7 +455,7 @@ Do this at least once after installing, and again after any change to the key or
@@ -448,7 +455,7 @@ Do this at least once after installing, and again after any change to the key or
| Command | What it does | Exit |
| Command | What it does | Exit |
|---|---|---|
|---|---|---|
-| `ovhost backup --all [--offsite] [--logical] [--no-prune] [--keep-daily n] [--keep-weekly n]` | back up every service with databases (verify pgBackRest, dump SQLite, dump PostgreSQL when due or `--logical`); prune; summary; optionally upload | 0 all ok · 1 lock/usage · 2 any failure |
+| `ovhost backup --all [--offsite] [--logical] [--no-prune] [--keep-daily n] [--keep-weekly n]` | back up every service with databases or object directories (verify pgBackRest, dump SQLite, dump PostgreSQL when due or `--logical`, archive object directories); prune; summary; optionally upload | 0 all ok · 1 lock/usage · 2 any failure |
| `ovhost offsite push [--run <run>]` | upload a run (default: the latest) again, for example after a failed upload | 0 · 1 config · 2 failure |
| `ovhost offsite push [--run <run>]` | upload a run (default: the latest) again, for example after a failed upload | 0 · 1 config · 2 failure |
| `ovhost offsite list [<service>] [--from-host <h>]` | runs off-host, newest first | 0 · 1 config · 2 bucket error |
| `ovhost offsite list [<service>] [--from-host <h>]` | runs off-host, newest first | 0 · 1 config · 2 bucket error |
| `ovhost offsite check` | config, key and bucket access; uploads nothing | 0 · 1 config · 2 bucket error |
| `ovhost offsite check` | config, key and bucket access; uploads nothing | 0 · 1 config · 2 bucket error |
@@ -92,6 +92,9 @@ block (see `host.example.json`):
@@ -92,6 +92,9 @@ block (see `host.example.json`):
`PRAGMA integrity_check`, run as the service user, must return `ok` for each copy;
`PRAGMA integrity_check`, run as the service user, must return `ok` for each copy;
otherwise the drill fails before anything starts. A PostgreSQL service restores its `.dump` into a
otherwise the drill fails before anything starts. A PostgreSQL service restores its `.dump` into a
scratch database instead — see [A PostgreSQL drill](#a-postgresql-drill).
scratch database instead — see [A PostgreSQL drill](#a-postgresql-drill).
+ Every `objects` directory the service declares is restored as well: its archive is extracted into
+ `<tmp>/<name>`, owned by the service user and never the live directory, and every blob is checked
+ against the sha256 in its file name.
2. **Start.** A second instance starts from the production checkout through `systemd-run`, as the
2. **Start.** A second instance starts from the production checkout through `systemd-run`, as the
service user. It uses the production unit's `ExecStart` and `WorkingDirectory`, the production
service user. It uses the production unit's `ExecStart` and `WorkingDirectory`, the production
unit's non-secret `Environment=` values and the production env file. After that it loads
unit's non-secret `Environment=` values and the production env file. After that it loads
@@ -184,7 +187,7 @@ Supported with declared overrides (the table above records which have passed on
@@ -184,7 +187,7 @@ Supported with declared overrides (the table above records which have passed on
- network
- network
- community
- community
- events
- events
-- host (the Host API: `host.db` only, counts only)
+- host (the Host API: the `ov_host` database and the tenant object store, whose blobs are sha256-checked; counts only)
- billing
- billing
- chat
- chat
- search
- search
@@ -285,6 +285,6 @@ the history of any site.
@@ -285,6 +285,6 @@ the history of any site.
| Unverified or lapsed custom domains reach nginx's default server | medium (confusion, mis-served content on a stranger's domain) | operator: catch-all `default_server` (docs/launch.md step 0) |
| Unverified or lapsed custom domains reach nginx's default server | medium (confusion, mis-served content on a stranger's domain) | operator: catch-all `default_server` (docs/launch.md step 0) |
| Cookie bomb across tenants and the dashboard (no PSL) | low (per-visitor nuisance; takedown exists) | owner decision (§5) |
| Cookie bomb across tenants and the dashboard (no PSL) | low (per-visitor nuisance; takedown exists) | owner decision (§5) |
| No abuse mailbox/owner | medium (process, not code) | owner |
| No abuse mailbox/owner | medium (process, not code) | owner |
-| Objects not backed up | medium for tenants (alpha, documented) | follow-up: add the object store to the backup run |
+| Objects not backed up | medium for tenants (alpha, documented) | fixed: the inventory's `objects` entry archives the store, the off-host copy encrypts it, and a drill restores it and checks every blob's sha256 |
| Decompressing an upload blocked the event loop (`gunzipSync`) | done: async `zlib.gunzip` on the libuv threadpool (`server/artifacts/archive.js`), same `maxOutputLength` cap | done |
| Decompressing an upload blocked the event loop (`gunzipSync`) | done: async `zlib.gunzip` on the libuv threadpool (`server/artifacts/archive.js`), same `maxOutputLength` cap | done |
| Takedown routes not in released `openvibe-contracts` `host.site.manage.implementedBy` | none at runtime (the contract check passes) | Contracts: next release (docs/launch.md) |
| Takedown routes not in released `openvibe-contracts` `host.site.manage.implementedBy` | none at runtime (the contract check passes) | Contracts: next release (docs/launch.md) |
@@ -940,7 +940,7 @@
@@ -940,7 +940,7 @@
}
}
},
},
"host": {
"host": {
- "_note": "OpenVibe.Host Stage B (tenant static hosting): the Host API service on port 4910. Not deployed as of 22 Sep 2026. Its content-addressed objects live in /var/lib/openvibe-host-api, which is NOT ovhost's own stateDir (/var/lib/openvibe-host); its database is the PostgreSQL database ov_host. Tenant vhosts: `ovhost nginx tenants host`.",
+ "_note": "OpenVibe.Host Stage B (tenant static hosting): the Host API service on port 4910. Not deployed as of 22 Sep 2026. Its content-addressed objects live in /var/lib/openvibe-host-api/objects (`HOST_STORAGE_DIR`), which is NOT ovhost's own stateDir (/var/lib/openvibe-host); its database is the PostgreSQL database ov_host. Tenant vhosts: `ovhost nginx tenants host`.",
"repo": "/opt/openvibe.host",
"repo": "/opt/openvibe.host",
"units": [
"units": [
"openvibe-host.service"
"openvibe-host.service"
@@ -976,6 +976,12 @@
@@ -976,6 +976,12 @@
"database": "ov_host"
"database": "ov_host"
}
}
],
],
+ "objects": [
+ {
+ "name": "objects",
+ "path": "/var/lib/openvibe-host-api/objects"
+ }
+ ],
"nginx": {
"nginx": {
"tenants": {
"tenants": {
"sitesDomain": "openvibe.host",
"sitesDomain": "openvibe.host",
@@ -1036,7 +1042,7 @@
@@ -1036,7 +1042,7 @@
"table": "host_domains"
"table": "host_domains"
}
}
],
],
- "_notes": "Restores the ov_host database only: `ovhost backup host` does not copy the content-addressed objects, so tenant content cannot be compared (every /api/v1 read needs auth). HOST_WORKER=off stops DNS rechecks and the object sweep."
+ "_notes": "Restores the ov_host database and the archived `objects` directory into {tmp}/objects (HOST_STORAGE_DIR), checking every blob's sha256; tenant content is still not compared (every /api/v1 read needs auth). HOST_WORKER=off stops DNS rechecks and the object sweep."
}
}
},
},
"billing": {
"billing": {
@@ -1,7 +1,8 @@
@@ -1,7 +1,8 @@
'use strict';
'use strict';
/**
/**
* ovhost backup <service>: an online copy of every database the inventory declares for the service,
* ovhost backup <service>: an online copy of every database the inventory declares for the service,
- * into <backupDir>/<service>/<YYYYMMDD-HHMMSS>/<name>.<ext>, recorded in
+ * and a tar archive of every content-addressed object directory it declares, into
+ * <backupDir>/<service>/<YYYYMMDD-HHMMSS>/<name>.<ext>, recorded in
* <stateDir>/backups/<service>.jsonl. SQLite is copied by a worker running as the service user (root
* <stateDir>/backups/<service>.jsonl. SQLite is copied by a worker running as the service user (root
* never opens a service database — see sqlite-worker.js) and checked with PRAGMA quick_check.
* never opens a service database — see sqlite-worker.js) and checked with PRAGMA quick_check.
* PostgreSQL is not dumped every night: pgBackRest holds the cluster's full + differential backups
* PostgreSQL is not dumped every night: pgBackRest holds the cluster's full + differential backups
@@ -30,10 +31,11 @@
@@ -30,10 +31,11 @@
* throws a BackupError carrying the record. A SQLite backup that found none of its databases is
* throws a BackupError carrying the record. A SQLite backup that found none of its databases is
* recorded as ok: false, empty: true and leaves no directory behind; PostgreSQL never sets empty.
* recorded as ok: false, empty: true and leaves no directory behind; PostgreSQL never sets empty.
*
*
- * ovhost backup --all (backupAll): every service that declares databases, one after another. A
- * failing service never stops the others. Afterwards each service's backups are pruned to the last
- * 7 daily and 4 weekly good ones (lib/retention.js), and a JSON summary of the run is written to
- * <stateDir>/backup-runs/<run>.json. With --offsite the run is then uploaded (lib/offsite.js).
+ * ovhost backup --all (backupAll): every service that declares databases or object directories, one
+ * after another. A failing service never stops the others. Afterwards each service's backups are
+ * pruned to the last 7 daily and 4 weekly good ones (lib/retention.js), and a JSON summary of the run
+ * is written to <stateDir>/backup-runs/<run>.json. With --offsite the run is then uploaded
+ * (lib/offsite.js).
*/
*/
const path = require('path');
const path = require('path');
const { service } = require('../inventory');
const { service } = require('../inventory');
@@ -46,6 +48,19 @@ const DAY_MS = 86400000;
@@ -46,6 +48,19 @@ const DAY_MS = 86400000;
// differentials: it exists for the off-site copy and per-service drills, not for durability.
// differentials: it exists for the off-site copy and per-service drills, not for durability.
const LOGICAL_DUMP_INTERVAL_DAYS = 7;
const LOGICAL_DUMP_INTERVAL_DAYS = 7;
const ARCHIVER_STATE = 'pgbackrest-archiver.json';
const ARCHIVER_STATE = 'pgbackrest-archiver.json';
+// Room a tar archive of an object directory needs beyond `du -sb` of its source: per-file tar headers
+// and a little gzip overhead for already-compressed blobs.
+const ARCHIVE_HEADROOM = 1024 * 1024;
+// Host stages an upload at <store root>/tmp/<random>.part and renames it into place, so the store is
+// never quiescent while it is tarred. These warnings are not failures: the scratch is excluded too.
+const BENIGN_TAR_RE = /file changed as we read it|file removed before we read it|Cannot stat: No such file or directory/i;
+
+/** GNU tar exits 1 when a file changed or vanished mid-read. That is not a failed archive when every
+ * warning is one of those; any other stderr line means the run failed. */
+function tarWarningsOnly(stderr) {
+ const lines = String(stderr || '').split('\n').map((l) => l.trim()).filter(Boolean);
+ return lines.length > 0 && lines.every((l) => BENIGN_TAR_RE.test(l));
+}
class BackupError extends Error {
class BackupError extends Error {
constructor(message, record) { super(message); this.record = record; }
constructor(message, record) { super(message); this.record = record; }
@@ -130,7 +145,7 @@ async function lastGoodDumpAt(exec, inv, id) {
@@ -130,7 +145,7 @@ async function lastGoodDumpAt(exec, inv, id) {
async function backup(ctx, id, { reason = 'manual', logical = false } = {}) {
async function backup(ctx, id, { reason = 'manual', logical = false } = {}) {
const { exec, inv, log } = ctx;
const { exec, inv, log } = ctx;
const svc = service(inv, id);
const svc = service(inv, id);
- if (!svc.databases.length) throw new Error(`${id} declares no databases`);
+ if (!svc.databases.length && !svc.objects.length) throw new Error(`${id} declares no databases or object directories`);
const postgres = svc.databases.filter((d) => d.engine === 'postgresql');
const postgres = svc.databases.filter((d) => d.engine === 'postgresql');
const hasPostgres = postgres.length > 0;
const hasPostgres = postgres.length > 0;
const engine = hasPostgres ? 'postgresql' : 'sqlite';
const engine = hasPostgres ? 'postgresql' : 'sqlite';
@@ -156,8 +171,9 @@ async function backup(ctx, id, { reason = 'manual', logical = false } = {}) {
@@ -156,8 +171,9 @@ async function backup(ctx, id, { reason = 'manual', logical = false } = {}) {
const at = last === null ? NaN : Date.parse(last);
const at = last === null ? NaN : Date.parse(last);
if (logical || !Number.isFinite(at) || exec.now() - at > LOGICAL_DUMP_INTERVAL_DAYS * DAY_MS) dump.push(...postgres);
if (logical || !Number.isFinite(at) || exec.now() - at > LOGICAL_DUMP_INTERVAL_DAYS * DAY_MS) dump.push(...postgres);
}
}
- // A verification-only run (PostgreSQL, no dump due, no SQLite database) leaves nothing on disk.
- if (!dump.length && !svc.databases.some((d) => d.engine === 'sqlite')) {
+ // A verification-only run (PostgreSQL, no dump due, no SQLite database, no objects) leaves
+ // nothing on disk.
+ if (!dump.length && !svc.databases.some((d) => d.engine === 'sqlite') && !svc.objects.length) {
const record = { service: id, at: new Date(exec.now()).toISOString(), reason, engine, ok: true, files: [], verified };
const record = { service: id, at: new Date(exec.now()).toISOString(), reason, engine, ok: true, files: [], verified };
await exec.appendFile(path.join(inv.stateDir, 'backups', `${id}.jsonl`), `${JSON.stringify(record)}\n`);
await exec.appendFile(path.join(inv.stateDir, 'backups', `${id}.jsonl`), `${JSON.stringify(record)}\n`);
return record;
return record;
@@ -211,9 +227,46 @@ async function backup(ctx, id, { reason = 'manual', logical = false } = {}) {
@@ -211,9 +227,46 @@ async function backup(ctx, id, { reason = 'manual', logical = false } = {}) {
}
}
files.push({ name: db.name, source: db.database, dest: path.join(dir, file), file, check: 'pg_dump -Fc', seconds: Math.round((exec.now() - t0) / 1000) });
files.push({ name: db.name, source: db.database, dest: path.join(dir, file), file, check: 'pg_dump -Fc', seconds: Math.round((exec.now() - t0) / 1000) });
}
}
- // Take the staging directory back first: after this the service user (or postgres) can no
- // longer reach anything inside it, so what is checked below cannot change underneath.
+ // Take the staging directory back FIRST: while the service user (or postgres) owns it, it could
+ // plant a symlink at the path root is about to write the object archive through, and root's
+ // `tar -czf` would follow it. Once root:root 0700, neither can reach it, and everything written
+ // below is checked before the move.
await lockDown(exec, stage, 0o700);
await lockDown(exec, stage, 0o700);
+ // Content-addressed object directories: one tar archive each, created by root AFTER the stage
+ // is root-locked. The drill extracts this archive into its own scratch directory and checks
+ // every blob's sha256; nothing here needs to open a blob individually.
+ for (const obj of svc.objects) {
+ const file = `${obj.name}.tar.gz`;
+ const archive = path.join(stage, file);
+ if (!(await exec.stat(obj.path))) { files.push({ name: obj.name, source: obj.path, skipped: 'not found' }); log(`skip ${obj.path}: not found`); continue; }
+ const t0 = exec.now();
+ // `du -sb` of the source: the uncompressed size the drill must find room for, and the
+ // figure the staging space check below is made against.
+ const du = await exec.run('du', ['-sb', '--', obj.path], { privileged: true });
+ const sourceBytes = du.code === 0 ? Number(String(du.stdout).trim().split(/\s+/)[0]) : null;
+ if (Number.isFinite(sourceBytes)) {
+ const free = (await exec.statfs(inv.backupStagingDir)).free;
+ const needed = sourceBytes + ARCHIVE_HEADROOM;
+ if (free < needed) {
+ files.push({ name: obj.name, source: obj.path, dest: path.join(dir, file), error: `not enough free space under ${inv.backupStagingDir}: ${free} bytes free, need ${needed} for ${obj.name}` });
+ log(`FAILED ${obj.path}: not enough free space under ${inv.backupStagingDir} (${free} < ${needed})`);
+ continue;
+ }
+ }
+ // Clear whatever might sit at the archive's path — a link planted while the service user
+ // owned the stage would survive lockDown — before root's tar opens it for writing.
+ await exec.run('rm', ['-rf', '--', archive], { privileged: true });
+ log(`backing up object directory ${obj.path} → ${path.join(dir, file)}`);
+ // Host's transient upload scratch lives at <root>/tmp/*.part; exclude it, and tolerate a
+ // tar exit of 1 whose only warnings are a file that changed or vanished mid-read.
+ const r = await exec.run('tar', ['-czf', archive, '-C', obj.path, '--exclude=./tmp', '--exclude=./tmp/*', '.'], { privileged: true });
+ if (r.code !== 0 && !(r.code === 1 && tarWarningsOnly(r.stderr))) {
+ files.push({ name: obj.name, source: obj.path, dest: path.join(dir, file), error: `tar: ${(r.stderr || '').trim().slice(0, 200)}` });
+ log(`FAILED ${obj.path}: ${(r.stderr || '').trim().slice(0, 200)}`);
+ continue;
+ }
+ files.push({ name: obj.name, source: obj.path, dest: path.join(dir, file), file, kind: 'objects', sourceBytes, check: 'object archive', seconds: Math.round((exec.now() - t0) / 1000) });
+ }
for (const f of files.filter((x) => x.file && !x.error)) {
for (const f of files.filter((x) => x.file && !x.error)) {
const p = path.join(stage, f.file);
const p = path.join(stage, f.file);
const fst = await exec.stat(p);
const fst = await exec.stat(p);
@@ -299,7 +352,7 @@ async function backupAll(ctx, { daily = 7, weekly = 4, prune: doPrune = true, of
@@ -299,7 +352,7 @@ async function backupAll(ctx, { daily = 7, weekly = 4, prune: doPrune = true, of
const file = path.join(inv.stateDir, 'backup-runs', `${run}.json`);
const file = path.join(inv.stateDir, 'backup-runs', `${run}.json`);
try {
try {
for (const svc of Object.values(inv.services)) {
for (const svc of Object.values(inv.services)) {
- if (!svc.databases.length) continue;
+ if (!svc.databases.length && !svc.objects.length) continue;
const entry = { service: svc.id, status: 'ok', dir: null, files: [], error: null, pruned: [] };
const entry = { service: svc.id, status: 'ok', dir: null, files: [], error: null, pruned: [] };
summary.services.push(entry);
summary.services.push(entry);
try {
try {
@@ -12,6 +12,9 @@
@@ -12,6 +12,9 @@
* postgres, verify it with `pg_restore --list`, then create a scratch database and
* postgres, verify it with `pg_restore --list`, then create a scratch database and
* a login role `ov_<id>_drill_<stamp>` and `pg_restore --no-owner --role=…` the
* a login role `ov_<id>_drill_<stamp>` and `pg_restore --no-owner --role=…` the
* archive into the scratch database. Both are dropped again in a finally.
* archive into the scratch database. Both are dropped again in a finally.
+ * Every object directory the service declares is also restored: its archive is extracted into
+ * <tmp>/<name> (owned by the service user, never the live directory) and every blob is checked
+ * against the sha256 in its file name.
* 2. Start a second instance from the production checkout, as the service user, through
* 2. Start a second instance from the production checkout, as the service user, through
* systemd-run. It uses the production unit's ExecStart and WorkingDirectory, the production
* systemd-run. It uses the production unit's ExecStart and WorkingDirectory, the production
* env file, and a second env file with the drill's overrides: its port, the restored database
* env file, and a second env file with the drill's overrides: its port, the restored database
@@ -43,6 +46,8 @@
@@ -43,6 +46,8 @@
* allows loopback).
* allows loopback).
*/
*/
const path = require('path');
const path = require('path');
+const crypto = require('crypto');
+const { pipeline } = require('stream/promises');
const { service } = require('../inventory');
const { service } = require('../inventory');
const envfile = require('../envfile');
const envfile = require('../envfile');
const systemd = require('../systemd');
const systemd = require('../systemd');
@@ -147,12 +152,83 @@ async function countRows(exec, db, table, as) {
@@ -147,12 +152,83 @@ async function countRows(exec, db, table, as) {
return n;
return n;
}
}
+// A content-addressed blob is named by its own sha256: that name is the whole integrity check.
+const BLOB_RE = /^[0-9a-f]{64}$/;
+
+/** Every non-directory entry under `dir`, recursively (regular files and anything else tar restored). */
+async function walkFiles(exec, dir) {
+ const out = [];
+ for (const e of (await exec.readdir(dir)) || []) {
+ const p = path.join(dir, e.name);
+ const st = await exec.stat(p);
+ if (!st) continue;
+ // Never follow a symlink: `exec.stat` follows it, so a restored member `x -> /` would otherwise
+ // be recursed into and walk (and hash) files outside the scratch directory as root. It is
+ // returned as a leaf so verifyObjectBlobs refuses it.
+ if (st.isDir && !st.isSymlink) out.push(...await walkFiles(exec, p));
+ else out.push(p);
+ }
+ return out;
+}
+
+/** The sha256 of a file, streamed so a large blob is never held in memory. */
+async function sha256Stream(exec, p) {
+ const hash = crypto.createHash('sha256');
+ await pipeline(exec.readStream(p), hash);
+ return hash.digest('hex');
+}
+
+/**
+ * Check a restored object directory: every regular file is a blob named by its own sha256. A file
+ * whose name is not a sha256 is not a blob (Host keeps transient write scratch under its store
+ * root) and is counted as skipped, never verified. Any symlink or special file (device, fifo,
+ * socket) in the restored tree is refused whatever its name, as is a blob-named entry that is not a
+ * plain file, as in a database copy. `entry` gets the counts and the sha256 mismatch list.
+ */
+async function verifyObjectBlobs(exec, dir, entry) {
+ const bad = [];
+ for (const p of await walkFiles(exec, dir)) {
+ const st = await exec.stat(p);
+ if (!st) continue;
+ const name = path.basename(p);
+ // walkFiles returns symlinks as leaves; a special file is neither a file nor a directory.
+ if (st.isSymlink || !st.isFile) { bad.push(`${path.relative(dir, p)} (symlink or special file)`); continue; }
+ if (!BLOB_RE.test(name)) { entry.skipped += 1; entry.bytes += st.size; continue; }
+ if (st.nlink !== 1) { bad.push(`${path.relative(dir, p)} (not a plain file)`); continue; }
+ entry.bytes += st.size;
+ const hash = await sha256Stream(exec, p);
+ if (hash !== name) { bad.push(`${path.relative(dir, p)} (sha256 ${hash})`); continue; }
+ entry.blobs += 1;
+ }
+ entry.mismatches = bad.length;
+ if (bad.length) throw new Error(`${bad.length} blob(s) do not match their name: ${bad.slice(0, 3).join(', ')}${bad.length > 3 ? ', …' : ''}`);
+}
+
+/**
+ * A `tar -tvzf` listing line: permissions, owner/group, size, date, time, name. Only a regular file
+ * ('-') or a directory ('d') may be extracted as root; a symlink, hard link or special file, or an
+ * absolute or `..` name, could write outside the scratch directory. -> the first offending member.
+ */
+function tarMemberProblem(listing) {
+ for (const line of String(listing || '').split('\n')) {
+ if (!line.trim()) continue;
+ const m = /^(\S+)\s+\S+\s+\d+\s+\S+\s+\S+\s+(.*)$/.exec(line);
+ if (!m) return `unparsable tar listing line: ${line.trim().slice(0, 80)}`;
+ const type = m[1][0];
+ const name = m[2].replace(/ -> .*$/, '').trim();
+ if (type !== '-' && type !== 'd') return `member ${name} is not a regular file or directory (${type})`;
+ if (name.startsWith('/') || name.split('/').includes('..')) return `member ${name} is not a safe relative path`;
+ }
+ return null;
+}
+
// ── inputs ───────────────────────────────────────────────────────────────────
// ── inputs ───────────────────────────────────────────────────────────────────
/** The backup to restore: --backup <dir>, else the newest usable record in <stateDir>/backups/<id>.jsonl. */
/** The backup to restore: --backup <dir>, else the newest usable record in <stateDir>/backups/<id>.jsonl. */
async function resolveBackup(exec, inv, svc, dirArg) {
async function resolveBackup(exec, inv, svc, dirArg) {
let dir = dirArg;
let dir = dirArg;
let takenAt = null;
let takenAt = null;
+ let record = null;
const restored = Object.keys(svc.drill.databases).map((name) => svc.databases.find((d) => d.name === name));
const restored = Object.keys(svc.drill.databases).map((name) => svc.databases.find((d) => d.name === name));
const pgTargets = restored.filter((d) => d.engine === 'postgresql');
const pgTargets = restored.filter((d) => d.engine === 'postgresql');
if (dir) {
if (dir) {
@@ -171,6 +247,7 @@ async function resolveBackup(exec, inv, svc, dirArg) {
@@ -171,6 +247,7 @@ async function resolveBackup(exec, inv, svc, dirArg) {
const last = usable[usable.length - 1];
const last = usable[usable.length - 1];
dir = last.dir;
dir = last.dir;
takenAt = last.at || null;
takenAt = last.at || null;
+ record = last;
}
}
const st = await exec.stat(dir);
const st = await exec.stat(dir);
if (!st || !st.isDir) throw new DrillError(`backup directory ${dir} not found`);
if (!st || !st.isDir) throw new DrillError(`backup directory ${dir} not found`);
@@ -184,7 +261,20 @@ async function resolveBackup(exec, inv, svc, dirArg) {
@@ -184,7 +261,20 @@ async function resolveBackup(exec, inv, svc, dirArg) {
if (db.engine === 'postgresql') files.push({ name, file, production: db, engine: 'postgresql', url: target.url, directUrl: target.directUrl });
if (db.engine === 'postgresql') files.push({ name, file, production: db, engine: 'postgresql', url: target.url, directUrl: target.directUrl });
else files.push({ name, file, production: db.path, envName: target.env, dir: target.dir, ext: path.extname(db.path) || '.db', engine: 'sqlite' });
else files.push({ name, file, production: db.path, envName: target.env, dir: target.dir, ext: path.extname(db.path) || '.db', engine: 'sqlite' });
}
}
- return { dir, takenAt, files };
+ // Object directories are restored from the same backup: the archive holds the declared
+ // directory's contents, so the drill must not run without it (there would be nothing to verify).
+ const objects = [];
+ for (const obj of svc.objects) {
+ const file = path.join(dir, `${obj.name}.tar.gz`);
+ const fst = await exec.stat(file);
+ if (!fst || !fst.isFile) throw new DrillError(`backup ${dir} has no copy of the ${obj.name} object directory (${path.basename(file)})`);
+ // The uncompressed size recorded at backup time (`du -sb`), so the drill can check it has room
+ // to extract. A --backup directory has no record; its compressed size is the best we can do.
+ const rec = record && (record.files || []).find((f) => f.file === `${obj.name}.tar.gz`);
+ const sourceBytes = rec && Number.isFinite(rec.sourceBytes) ? rec.sourceBytes : fst.size;
+ objects.push({ name: obj.name, file, production: obj.path, sourceBytes });
+ }
+ return { dir, takenAt, files, objects };
}
}
/** How production starts the service: ExecStart, WorkingDirectory and non-secret Environment=. */
/** How production starts the service: ExecStart, WorkingDirectory and non-secret Environment=. */
@@ -287,6 +377,7 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
@@ -287,6 +377,7 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
unit: unitName,
unit: unitName,
pid: null,
pid: null,
databases: [],
databases: [],
+ objects: [],
ready: null,
ready: null,
compare: [],
compare: [],
counts: [],
counts: [],
@@ -322,6 +413,15 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
@@ -322,6 +413,15 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
for (const name of left.databases) { log(`dropping a scratch database an earlier drill left: ${name}`); await dbengine.dropDatabase(exec, name); }
for (const name of left.databases) { log(`dropping a scratch database an earlier drill left: ${name}`); await dbengine.dropDatabase(exec, name); }
for (const name of left.roles) { log(`dropping a scratch role an earlier drill left: ${name}`); await dbengine.dropRole(exec, name); }
for (const name of left.roles) { log(`dropping a scratch role an earlier drill left: ${name}`); await dbengine.dropRole(exec, name); }
}
}
+ // Free space for the object archives too: extracting one writes its whole uncompressed
+ // contents under the drill directory. The size `du -sb` measured at backup time travels in the
+ // backup record; a --backup directory falls back to the archive's compressed size.
+ const objectBytes = bk.objects.reduce((n, o) => n + (o.sourceBytes || 0), 0);
+ if (objectBytes) {
+ await exec.mkdir(inv.drillDir, { mode: 0o711 });
+ const free = (await exec.statfs(inv.drillDir)).free;
+ if (free < objectBytes) throw fail('space', `not enough free space under ${inv.drillDir}: ${free} bytes free, need ${objectBytes} (the object directories' contents)`);
+ }
if (await exec.stat(tmp)) throw fail('prepare', `${tmp} already exists`);
if (await exec.stat(tmp)) throw fail('prepare', `${tmp} already exists`);
await exec.mkdir(inv.drillDir, { mode: 0o711 });
await exec.mkdir(inv.drillDir, { mode: 0o711 });
// postgres reads the dump copy in <tmp>/pg, so it must be able to traverse <tmp> (0711: no listing;
// postgres reads the dump copy in <tmp>/pg, so it must be able to traverse <tmp> (0711: no listing;
@@ -385,6 +485,29 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
@@ -385,6 +485,29 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
if (entry.integrity !== 'ok') throw fail('integrity', `${f.name}: integrity_check = ${entry.integrity.slice(0, 300)}`);
if (entry.integrity !== 'ok') throw fail('integrity', `${f.name}: integrity_check = ${entry.integrity.slice(0, 300)}`);
log(`integrity_check ${f.name}: ok`);
log(`integrity_check ${f.name}: ok`);
}
}
+ // 1b. restore every declared object directory and check every blob's sha256. The archive is
+ // root-only, so root extracts it into the drill's own scratch directory — never over the live
+ // object directory, whose path reaches the drill instance only through its env override.
+ for (const obj of bk.objects) {
+ const dest = path.join(tmp, obj.name);
+ await exec.mkdir(dest, { owner: svc.runAs, mode: 0o700 });
+ // List the archive before extracting as root: only regular files and directories, with
+ // relative names, may be written. `--no-same-owner --no-same-permissions` then keeps the
+ // archive from restoring production's ownership or modes onto the drill copy.
+ const list = await exec.run('tar', ['-tvzf', obj.file], { privileged: true, env: { LC_ALL: 'C' } });
+ if (list.code !== 0) throw fail('objects', `listing ${obj.file} failed: ${(list.stderr || '').trim().slice(0, 200)}`);
+ if (/Removing leading|Member name contains '\.\.'/.test(list.stderr || '')) throw fail('objects', `${obj.file}: the archive holds an absolute or '..' member`);
+ const problem = tarMemberProblem(list.stdout);
+ if (problem) throw fail('objects', `${obj.file}: ${problem}`);
+ const x = await exec.run('tar', ['-xzf', obj.file, '-C', dest, '--no-same-owner', '--no-same-permissions'], { privileged: true });
+ if (x.code !== 0) throw fail('objects', `extracting ${obj.file} failed: ${(x.stderr || '').trim().slice(0, 200)}`);
+ const entry = { name: obj.name, source: obj.file, copy: dest, blobs: 0, skipped: 0, bytes: 0, mismatches: 0, integrity: null };
+ record.objects.push(entry);
+ log(`restoring object directory ${obj.file} → ${dest} (${svc.runAs})`);
+ try { await verifyObjectBlobs(exec, dest, entry); } catch (err) { entry.integrity = `error: ${err.message}`; throw fail('objects', `${obj.name}: ${err.message}`); }
+ entry.integrity = 'sha256 ok';
+ log(`object directory ${obj.name}: ${entry.blobs} blob(s) sha256 ok${entry.skipped ? `, ${entry.skipped} non-blob file(s) skipped` : ''}`);
+ }
const restoredCounts = [];
const restoredCounts = [];
for (const c of d.counts) {
for (const c of d.counts) {
const rdb = svc.databases.find((x) => x.name === c.db);
const rdb = svc.databases.find((x) => x.name === c.db);
@@ -589,6 +712,9 @@ function markdownRow(r) {
@@ -589,6 +712,9 @@ function markdownRow(r) {
for (const db of r.databases) checks.push(db.engine === 'postgresql'
for (const db of r.databases) checks.push(db.engine === 'postgresql'
? `\`pg_restore --list\` (${db.name}) = ${db.integrity === 'pg_restore --list ok' ? 'ok' : cell(db.integrity).slice(0, 80)}`
? `\`pg_restore --list\` (${db.name}) = ${db.integrity === 'pg_restore --list ok' ? 'ok' : cell(db.integrity).slice(0, 80)}`
: `\`pragma integrity_check\` (${db.name}) = ${db.integrity === 'ok' ? 'ok' : cell(db.integrity).slice(0, 80)}`);
: `\`pragma integrity_check\` (${db.name}) = ${db.integrity === 'ok' ? 'ok' : cell(db.integrity).slice(0, 80)}`);
+ for (const o of r.objects || []) checks.push(o.integrity === 'sha256 ok'
+ ? `object store \`${o.name}\` ${o.blobs} blob(s) sha256 ok`
+ : `object store \`${o.name}\`: ${cell(o.integrity).slice(0, 80)}`);
if (r.ready) checks.push(r.ready.ok ? `drill instance \`${new URL(r.ready.url).pathname}\` ${r.ready.status} after ${r.ready.seconds}s` : `drill instance not ready (${r.ready.status || 'no answer'})`);
if (r.ready) checks.push(r.ready.ok ? `drill instance \`${new URL(r.ready.url).pathname}\` ${r.ready.status} after ${r.ready.seconds}s` : `drill instance not ready (${r.ready.status || 'no answer'})`);
const same = r.compare.filter((c) => c.match).map((c) => `\`${c.path}\``);
const same = r.compare.filter((c) => c.match).map((c) => `\`${c.path}\``);
const diff = r.compare.filter((c) => !c.match).map((c) => `\`${c.path}\` differs (${cell(c.detail)})`);
const diff = r.compare.filter((c) => !c.match).map((c) => `\`${c.path}\` differs (${cell(c.detail)})`);
@@ -603,4 +729,4 @@ function markdownRow(r) {
@@ -603,4 +729,4 @@ function markdownRow(r) {
return `| ${when} | ${r.service} | \`${r.backup}/\` (\`ovhost drill ${r.service}\`) | ${checks.join('; ')} | ${r.result} |`;
return `| ${when} | ${r.service} | \`${r.backup}/\` (\`ovhost drill ${r.service}\`) | ${checks.join('; ')} | ${r.result} |`;
}
}
-module.exports = { drill, DrillError, compareBodies };
+module.exports = { drill, DrillError, compareBodies, verifyObjectBlobs, tarMemberProblem };
@@ -143,6 +143,26 @@ function normalisePreflight(raw, where) {
@@ -143,6 +143,26 @@ function normalisePreflight(raw, where) {
return out;
return out;
}
}
+/**
+ * `objects`: directories of content-addressed blobs the service owns OUTSIDE its databases, such as
+ * OpenVibe.Host's tenant object store (host.example.json: `HOST_STORAGE_DIR`). Each regular file in
+ * the directory is named by its own sha256, so `ovhost backup` can archive the tree as one artifact
+ * and a drill can check every blob's sha256 after restoring it.
+ */
+function normaliseObjects(raw, where) {
+ if (raw == null) return [];
+ if (!Array.isArray(raw)) throw new InventoryError(`${where}.objects must be a list of directories`);
+ const seen = new Set();
+ return raw.map((o, i) => {
+ const at = `${where}.objects[${i}]`;
+ if (!o || typeof o !== 'object' || Array.isArray(o)) throw new InventoryError(`${at} must be an object`);
+ if (!FILE_NAME_RE.test(o.name || '')) throw new InventoryError(`${at}.name must be a plain name`);
+ if (seen.has(o.name)) throw new InventoryError(`${at}.name "${o.name}" is listed twice`);
+ seen.add(o.name);
+ return { name: o.name, path: abs(o.path, `${at}.path`) };
+ });
+}
+
function normaliseService(id, raw, defaults) {
function normaliseService(id, raw, defaults) {
if (!ID_RE.test(id)) throw new InventoryError(`service id "${id}" must match ${ID_RE}`);
if (!ID_RE.test(id)) throw new InventoryError(`service id "${id}" must match ${ID_RE}`);
const where = `services.${id}`;
const where = `services.${id}`;
@@ -236,6 +256,9 @@ function normaliseService(id, raw, defaults) {
@@ -236,6 +256,9 @@ function normaliseService(id, raw, defaults) {
}
}
throw new InventoryError(`${at}.engine must be sqlite or postgresql`);
throw new InventoryError(`${at}.engine must be sqlite or postgresql`);
}),
}),
+ // Content-addressed object directories alongside the databases (tenant objects, Host's
+ // HOST_STORAGE_DIR). Copied as one archive per directory; never opened as a database.
+ objects: normaliseObjects(s.objects, where),
backupOnChange: s.backupOnChange || [],
backupOnChange: s.backupOnChange || [],
nginx: s.nginx || null,
nginx: s.nginx || null,
// The public origin release notifications name (lib/announce.js); default: the manifest's publicOrigin.
// The public origin release notifications name (lib/announce.js); default: the manifest's publicOrigin.
@@ -1,10 +1,10 @@
@@ -1,10 +1,10 @@
'use strict';
'use strict';
/**
/**
- * Off-host copies of `ovhost backup --all` runs: every backed-up database is encrypted on the host
- * (lib/backup-crypto.js, AES-256-GCM with a key that stays on the host) and uploaded to S3-compatible
- * object storage (Backblaze B2 in production, the same API Media uses). Nothing is uploaded in the
- * clear except the run manifest, which holds names, sizes and hashes, never data or secrets, and
- * carries an HMAC made with the backup key.
+ * Off-host copies of `ovhost backup --all` runs: every backed-up file (a database copy or an object
+ * archive) is encrypted on the host (lib/backup-crypto.js, AES-256-GCM with a key that stays on the
+ * host) and uploaded to S3-compatible object storage (Backblaze B2 in production, the same API Media
+ * uses). Nothing is uploaded in the clear except the run manifest, which holds names, sizes and hashes,
+ * never data or secrets, and carries an HMAC made with the backup key.
*
*
* Configuration comes from a root-only env file (default /etc/openvibe/backup.env):
* Configuration comes from a root-only env file (default /etc/openvibe/backup.env):
*
*
@@ -252,7 +252,7 @@ async function push(ctx, summary, { cfg, client, partSize = PART_SIZE, chunkSize
@@ -252,7 +252,7 @@ async function push(ctx, summary, { cfg, client, partSize = PART_SIZE, chunkSize
await pipeline(exec.readStream(f.dest), plain, bcrypto.createEncryptStream(master, chunkSize ? { chunkSize } : {}), cipher, async (source) => { up = await uploadIterable(client, cfg.bucket, key, source, { partSize }); });
await pipeline(exec.readStream(f.dest), plain, bcrypto.createEncryptStream(master, chunkSize ? { chunkSize } : {}), cipher, async (source) => { up = await uploadIterable(client, cfg.bucket, key, source, { partSize }); });
const p = plain.result();
const p = plain.result();
const c = cipher.result();
const c = cipher.result();
- const entry = { service: svc.service, name: f.name, file: name, object: key, source: f.source || null, plainBytes: p.bytes, plainSha256: p.sha256, cipherBytes: c.bytes, cipherSha256: c.sha256, parts: up.parts, seconds: Math.round((exec.now() - t0) / 1000) };
+ const entry = { service: svc.service, name: f.name, kind: f.kind || 'database', file: name, object: key, source: f.source || null, plainBytes: p.bytes, plainSha256: p.sha256, cipherBytes: c.bytes, cipherSha256: c.sha256, parts: up.parts, seconds: Math.round((exec.now() - t0) / 1000) };
result.uploaded.push(entry);
result.uploaded.push(entry);
(manifest.services[svc.service] = manifest.services[svc.service] || { files: [] }).files.push(entry);
(manifest.services[svc.service] = manifest.services[svc.service] || { files: [] }).files.push(entry);
log(`uploaded ${f.dest} → s3://${cfg.bucket}/${key} (${c.bytes} bytes encrypted)`);
log(`uploaded ${f.dest} → s3://${cfg.bucket}/${key} (${c.bytes} bytes encrypted)`);
@@ -322,7 +322,7 @@ async function readManifest(client, cfg, master, host, run) {
@@ -322,7 +322,7 @@ async function readManifest(client, cfg, master, host, run) {
return m;
return m;
}
}
-/** Directories restore-download must never write into: anything holding a declared database. */
+/** Directories restore-download must never write into: anything holding a declared database or object directory. */
function unsafeOut(inv, out) {
function unsafeOut(inv, out) {
for (const svc of Object.values(inv.services)) {
for (const svc of Object.values(inv.services)) {
for (const db of svc.databases) {
for (const db of svc.databases) {
@@ -331,6 +331,11 @@ function unsafeOut(inv, out) {
@@ -331,6 +331,11 @@ function unsafeOut(inv, out) {
const dir = path.dirname(db.path);
const dir = path.dirname(db.path);
if (out === dir || db.path.startsWith(`${out}/`) || out.startsWith(`${dir}/`)) return `${out} holds or sits inside ${svc.id}'s database directory ${dir}`;
if (out === dir || db.path.startsWith(`${out}/`) || out.startsWith(`${dir}/`)) return `${out} holds or sits inside ${svc.id}'s database directory ${dir}`;
}
}
+ // An object directory is itself a directory the live service writes: never restore over it.
+ for (const obj of svc.objects || []) {
+ const dir = obj.path;
+ if (out === dir || dir.startsWith(`${out}/`) || out.startsWith(`${dir}/`)) return `${out} holds or sits inside ${svc.id}'s object directory ${dir}`;
+ }
if (out === svc.repo || out.startsWith(`${svc.repo}/`) || svc.repo.startsWith(`${out}/`)) return `${out} overlaps ${svc.id}'s checkout ${svc.repo}`;
if (out === svc.repo || out.startsWith(`${svc.repo}/`) || svc.repo.startsWith(`${out}/`)) return `${out} overlaps ${svc.id}'s checkout ${svc.repo}`;
}
}
for (const d of [inv.backupDir, inv.stateDir]) if (out === d || d.startsWith(`${out}/`) || out.startsWith(`${d}/`)) return `${out} overlaps ${d}`;
for (const d of [inv.backupDir, inv.stateDir]) if (out === d || d.startsWith(`${out}/`) || out.startsWith(`${d}/`)) return `${out} overlaps ${d}`;
@@ -390,18 +395,25 @@ async function restoreDownload(ctx, id, runArg, { cfg, client, out = null, host:
@@ -390,18 +395,25 @@ async function restoreDownload(ctx, id, runArg, { cfg, client, out = null, host:
}
}
// A PostgreSQL artifact is a pg_dump archive, checked by parsing it (pg_restore --list, which
// A PostgreSQL artifact is a pg_dump archive, checked by parsing it (pg_restore --list, which
// needs no database and is run as the caller — the copy is root-only 0600, and postgres could
// needs no database and is run as the caller — the copy is root-only 0600, and postgres could
- // not read it); a SQLite copy is quick_checked through the worker.
+ // not read it); a SQLite copy is quick_checked through the worker. An object-directory archive
+ // is neither: its own sha256 has already matched the manifest, and a drill is what extracts it
+ // and checks every blob.
let check;
let check;
const isDump = path.extname(f.file) === '.dump';
const isDump = path.extname(f.file) === '.dump';
- try {
- if (isDump) { await dbengine.verifyDump(exec, dest, { as: null }); check = 'pg_restore --list ok'; }
- else {
- const rows = await exec.sqlite(dest, 'PRAGMA quick_check');
- check = rows && rows[0] ? String(Object.values(rows[0])[0]) : 'no answer';
- }
- } catch (err) { check = `error: ${err.message}`; }
- record.files.push({ name: f.name, file: dest, source: f.source, bytes: p.bytes, sha256: p.sha256, quickCheck: check });
- if (isDump ? check.startsWith('error') : check !== 'ok') throw new OffsiteError(`${dest}: ${check}`);
+ const isObjects = f.kind === 'objects' || f.file.endsWith('.tar.gz');
+ if (isObjects) {
+ check = 'object archive (verify with a drill)';
+ } else {
+ try {
+ if (isDump) { await dbengine.verifyDump(exec, dest, { as: null }); check = 'pg_restore --list ok'; }
+ else {
+ const rows = await exec.sqlite(dest, 'PRAGMA quick_check');
+ check = rows && rows[0] ? String(Object.values(rows[0])[0]) : 'no answer';
+ }
+ } catch (err) { check = `error: ${err.message}`; }
+ }
+ record.files.push({ name: f.name, kind: isObjects ? 'objects' : 'database', file: dest, source: f.source, bytes: p.bytes, sha256: p.sha256, quickCheck: isObjects ? null : check });
+ if (!isObjects && (isDump ? check.startsWith('error') : check !== 'ok')) throw new OffsiteError(`${dest}: ${check}`);
}
}
return record;
return record;
}
}
@@ -15,10 +15,12 @@ const retention = require('../lib/retention');
@@ -15,10 +15,12 @@ const retention = require('../lib/retention');
const offsite = require('../lib/offsite');
const offsite = require('../lib/offsite');
const { normalise } = require('../lib/inventory');
const { normalise } = require('../lib/inventory');
const { mockS3 } = require('./s3-mock');
const { mockS3 } = require('./s3-mock');
+const { readTarGz } = require('../lib/tar');
const DAY = 86400000;
const DAY = 86400000;
const KEY_HEX = crypto.createHash('sha256').update('test backup key').digest('hex');
const KEY_HEX = crypto.createHash('sha256').update('test backup key').digest('hex');
const S3_SECRET = `${SECRET}-s3`;
const S3_SECRET = `${SECRET}-s3`;
+const TENANT_BLOB = 'tenant object bytes\n';
/** The standard fake host plus AI (two databases) and OpenRe (not deployed), a backup env and key. */
/** The standard fake host plus AI (two databases) and OpenRe (not deployed), a backup env and key. */
function backupHost({ envMode = 0o600, keyMode = 0o600, envOwner = 'root', env = null } = {}) {
function backupHost({ envMode = 0o600, keyMode = 0o600, envOwner = 'root', env = null } = {}) {
@@ -75,6 +77,25 @@ function noSecrets(text, what) {
@@ -75,6 +77,25 @@ function noSecrets(text, what) {
for (const s of [S3_SECRET, SECRET, KEY_HEX]) assert.ok(!String(text).includes(s), `${what} leaked a secret value`);
for (const s of [S3_SECRET, SECRET, KEY_HEX]) assert.ok(!String(text).includes(s), `${what} leaked a secret value`);
}
}
+/** backupHost plus a service "host" on PostgreSQL with a declared content-addressed object directory. */
+function objectsHost() {
+ const host = backupHost();
+ const doc = JSON.parse(host.read('/etc/openvibe/host.json'));
+ doc.services.host = {
+ repo: '/opt/openvibe.host',
+ units: [],
+ databases: [{ name: 'host', engine: 'postgresql', database: 'ov_host' }],
+ objects: [{ name: 'objects', path: '/var/lib/openvibe-host-api/objects' }],
+ };
+ host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });
+ host.inv = normalise(doc);
+ host.pgDatabases.add('ov_host');
+ host.pgBackups = [{ type: 'diff', timestamp: { stop: Math.floor(host.exec.now() / 1000) - 3600 } }];
+ host.tenantSha = crypto.createHash('sha256').update(TENANT_BLOB).digest('hex');
+ host.put(`/var/lib/openvibe-host-api/objects/projects/prj_1/${host.tenantSha.slice(0, 2)}/${host.tenantSha}`, TENANT_BLOB, { owner: 'ubuntu', mode: 0o644 });
+ return host;
+}
+
async function decrypt(key, buf) {
async function decrypt(key, buf) {
const chunks = [];
const chunks = [];
await pipeline(Readable.from([buf]), bcrypto.createDecryptStream(key), async (src) => { for await (const c of src) chunks.push(c); });
await pipeline(Readable.from([buf]), bcrypto.createDecryptStream(key), async (src) => { for await (const c of src) chunks.push(c); });
@@ -355,6 +376,83 @@ runTests([
@@ -355,6 +376,83 @@ runTests([
assert.strictEqual(s3.objects.size, 0);
assert.strictEqual(s3.objects.size, 0);
}),
}),
+ test('--offsite: a declared object directory is archived, encrypted, uploaded and recorded in the manifest', async () => {
+ const host = objectsHost();
+ const r = await host.cli('backup', '--all', '--offsite');
+ assert.strictEqual(r.code, 0, r.out);
+ // The archive is an ordinary root-only copy in the local backup...
+ const local = host.files.get('/var/backups/openvibe/host/20260922-120000/objects.tar.gz');
+ assert.deepStrictEqual([local.owner, local.mode], ['root', 0o600]);
+ // ...and an encrypted object off-host, under the same rules as every other file.
+ const base = 'openvibe-backups/fake-host/20260922-120000/';
+ const key = `${base}host/objects.tar.gz.ovbk`;
+ assert.ok(host.s3.objects.has(key), [...host.s3.objects.keys()].join('\n'));
+ const enc = host.s3.objects.get(key);
+ assert.ok(!enc.includes('tenant object'), 'the archive is not encrypted');
+ const plain = await decrypt(Buffer.from(KEY_HEX, 'hex'), enc);
+ const entries = readTarGz(plain);
+ assert.deepStrictEqual(entries.map((e) => e.name), [`projects/prj_1/${host.tenantSha.slice(0, 2)}/${host.tenantSha}`]);
+ assert.strictEqual(entries[0].content, TENANT_BLOB);
+ const m = JSON.parse(host.s3.objects.get(`${base}manifest.json`));
+ const f = m.services.host.files.find((x) => x.file === 'objects.tar.gz');
+ assert.deepStrictEqual([f.kind, f.name], ['objects', 'objects']);
+ assert.strictEqual(f.plainSha256, crypto.createHash('sha256').update(plain).digest('hex'));
+ noSecrets(host.s3.objects.get(`${base}manifest.json`), 'the manifest');
+ noSecrets(JSON.stringify(host.s3.sent), 'S3 requests');
+ noSecrets(r.out, 'CLI output');
+ }),
+
+ test('an object archive is written only after root has taken the staging directory over', async () => {
+ const host = objectsHost();
+ const r = await host.cli('backup', 'host');
+ assert.strictEqual(r.code, 0, r.out);
+ const tarCall = host.calls.find((c) => c.cmd === 'tar' && c.args.some((a) => String(a).endsWith('objects.tar.gz')));
+ assert.ok(tarCall, 'the object directory was tarred');
+ assert.strictEqual(tarCall.privileged, true);
+ const stage = path.dirname(tarCall.args[1]);
+ const lockAt = host.calls.findIndex((c) => c.cmd === 'chown' && c.args[c.args.length - 1] === stage && c.args.includes('root:root'));
+ const chmodAt = host.calls.findIndex((c) => c.cmd === 'chmod' && c.args[c.args.length - 1] === stage && c.args.includes('0700'));
+ const tarAt = host.calls.indexOf(tarCall);
+ assert.ok(lockAt >= 0 && chmodAt >= 0, 'root locked the staging directory down');
+ assert.ok(lockAt < tarAt && chmodAt < tarAt, 'root owned the stage before tar opened the archive');
+ }),
+
+ test('a tar warning for transient scratch under the store is accepted, and the scratch excluded', async () => {
+ const host = objectsHost();
+ host.put('/var/lib/openvibe-host-api/objects/tmp/aaaaaaaaaaaa.part', 'half-written upload', { owner: 'ubuntu', mode: 0o644 });
+ host.onTar = (args) => (String(args[0]).includes('c') ? { code: 1, stderr: 'tar: ./tmp/aaaaaaaaaaaa.part: file changed as we read it\n' } : undefined);
+ const r = await host.cli('backup', 'host', '--json');
+ host.onTar = null;
+ assert.strictEqual(r.code, 0, r.out);
+ const rec = JSON.parse(r.out);
+ const f = rec.files.find((x) => x.kind === 'objects');
+ assert.ok(f && !f.error, JSON.stringify(f));
+ assert.ok(Number.isFinite(f.sourceBytes) && f.sourceBytes > 0, 'the source size was recorded for the drill');
+ const tarCall = host.calls.find((c) => c.cmd === 'tar' && c.args.some((a) => String(a).endsWith('.tar.gz')));
+ assert.ok(tarCall.args.includes('--exclude=./tmp'), tarCall.args.join(' '));
+ const names = readTarGz(host.files.get(`${rec.dir}/objects.tar.gz`).content).map((e) => e.name);
+ assert.ok(!names.some((n) => n.startsWith('tmp/')), `scratch was archived: ${names.join(', ')}`);
+ assert.ok(names.length >= 1, 'the tenant blob is still archived');
+ }),
+
+ test('a genuine tar failure still fails the backup', async () => {
+ const host = objectsHost();
+ host.onTar = (args) => (String(args[0]).includes('c') ? { code: 2, stderr: 'tar: ./projects: Cannot open: Permission denied\n' } : undefined);
+ const r = await host.cli('backup', 'host');
+ host.onTar = null;
+ assert.strictEqual(r.code, 2, r.out);
+ assert.match(r.out, /Permission denied/);
+ }),
+
+ test('an object directory is not archived when the staging filesystem has no room', async () => {
+ const host = objectsHost();
+ host.statfsFree = 1024;
+ const r = await host.cli('backup', 'host');
+ assert.strictEqual(r.code, 2, r.out);
+ assert.match(r.out, /not enough free space under \/var\/backups\/openvibe\.staging/);
+ assert.ok(!host.calls.some((c) => c.cmd === 'tar'), 'nothing was tarred');
+ }),
+
test('off-host retention: runs older than 30 days are deleted, the newest 7 always stay, and nothing is pruned after a failed upload', async () => {
test('off-host retention: runs older than 30 days are deleted, the newest 7 always stay, and nothing is pruned after a failed upload', async () => {
const host = backupHost();
const host = backupHost();
const put = (run) => {
const put = (run) => {
@@ -487,6 +585,22 @@ runTests([
@@ -487,6 +585,22 @@ runTests([
assert.match(refused.out, /pg_restore --list .* failed|text format dump/);
assert.match(refused.out, /pg_restore --list .* failed|text format dump/);
}),
}),
+ test('restore-download: an object archive is written without a database check and never over the live object directory', async () => {
+ const host = objectsHost();
+ await host.cli('backup', '--all', '--offsite');
+ const near = await host.cli('restore-download', 'host', 'latest', '--out', '/var/lib/openvibe-host-api/objects');
+ assert.strictEqual(near.code, 1, near.out);
+ assert.match(near.out, /refusing --out: .*object directory/);
+ const r = await host.cli('restore-download', 'host', 'latest', '--out', '/root/r-host', '--json');
+ assert.strictEqual(r.code, 0, r.out);
+ const rec = JSON.parse(r.out);
+ const f = rec.files.find((x) => x.file.endsWith('objects.tar.gz'));
+ assert.deepStrictEqual([f.kind, f.quickCheck], ['objects', null], 'an object archive is not quick_checked as a database');
+ assert.ok(host.files.has('/root/r-host/objects.tar.gz'));
+ assert.ok(!host.sqliteCalls.some((c) => c.db === '/root/r-host/objects.tar.gz'));
+ assert.ok(rec.files.some((x) => x.file.endsWith('host.dump') && x.kind === 'database'), 'the database is still checked');
+ }),
+
// ── configuration ──────────────────────────────────────────────────────────
// ── configuration ──────────────────────────────────────────────────────────
test('config: root-only env file and key; missing names listed by name; not root refused; no value ever printed', async () => {
test('config: root-only env file and key; missing names listed by name; not root refused; no value ever printed', async () => {
let host = backupHost({ envMode: 0o644 });
let host = backupHost({ envMode: 0o644 });
@@ -6,16 +6,24 @@
@@ -6,16 +6,24 @@
* the drill log is written.
* the drill log is written.
*/
*/
const assert = require('assert');
const assert = require('assert');
+const crypto = require('crypto');
const fs = require('fs');
const fs = require('fs');
const path = require('path');
const path = require('path');
const { scenario, test, runTests, SECRET } = require('./helpers');
const { scenario, test, runTests, SECRET } = require('./helpers');
const { normalise } = require('../lib/inventory');
const { normalise } = require('../lib/inventory');
-const { compareBodies } = require('../lib/commands/drill');
+const { compareBodies, verifyObjectBlobs } = require('../lib/commands/drill');
const DRILL_DIR = '/var/lib/openvibe-drills';
const DRILL_DIR = '/var/lib/openvibe-drills';
const PROD_DB = '/var/lib/openvibe-tools/tools.db';
const PROD_DB = '/var/lib/openvibe-tools/tools.db';
+const OBJECTS_DIR = '/var/lib/openvibe-host-api/objects';
const PASTES = '[{"id":"pst_1","title":"hello"},{"id":"pst_2","title":"world"}]';
const PASTES = '[{"id":"pst_1","title":"hello"},{"id":"pst_2","title":"world"}]';
+/** A content-addressed blob under the Host object-store layout: <root>/projects/<prj>/<aa>/<sha256>. */
+function blob(root, data) {
+ const sha = crypto.createHash('sha256').update(data).digest('hex');
+ return { sha, path: `${root}/projects/prj_1/${sha.slice(0, 2)}/${sha}` };
+}
+
function toolsEntry(drillOverrides = {}) {
function toolsEntry(drillOverrides = {}) {
return {
return {
repo: '/opt/openvibe.tools',
repo: '/opt/openvibe.tools',
@@ -40,10 +48,12 @@ function toolsEntry(drillOverrides = {}) {
@@ -40,10 +48,12 @@ function toolsEntry(drillOverrides = {}) {
* The standard fake host plus Tools in production, with one `ovhost backup tools` taken.
* The standard fake host plus Tools in production, with one `ovhost backup tools` taken.
* The drill instance's behaviour is controlled through `host.drill`.
* The drill instance's behaviour is controlled through `host.drill`.
*/
*/
-async function drillScenario({ drill: drillOverrides, backup = true } = {}) {
+async function drillScenario({ drill: drillOverrides, backup = true, objects = null, blobs = [] } = {}) {
const host = scenario();
const host = scenario();
const doc = JSON.parse(host.read('/etc/openvibe/host.json'));
const doc = JSON.parse(host.read('/etc/openvibe/host.json'));
- doc.services.tools = toolsEntry(drillOverrides);
+ const tools = toolsEntry(drillOverrides);
+ if (objects) tools.objects = objects;
+ doc.services.tools = tools;
host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });
host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });
const repo = host.createRepo('/opt/openvibe.tools', { owner: 'ubuntu' });
const repo = host.createRepo('/opt/openvibe.tools', { owner: 'ubuntu' });
@@ -100,6 +110,8 @@ async function drillScenario({ drill: drillOverrides, backup = true } = {}) {
@@ -100,6 +110,8 @@ async function drillScenario({ drill: drillOverrides, backup = true } = {}) {
return undefined;
return undefined;
};
};
+ for (const b of blobs) host.put(b.path, b.content, { owner: 'ubuntu', mode: 0o644 });
+
if (backup) {
if (backup) {
const b = await host.cli('backup', 'tools', '--json');
const b = await host.cli('backup', 'tools', '--json');
assert.strictEqual(b.code, 0, b.out);
assert.strictEqual(b.code, 0, b.out);
@@ -545,6 +557,137 @@ runTests([
@@ -545,6 +557,137 @@ runTests([
normalise({ services: { apps: multi } }); // valid inventory; the drill itself refuses (below)
normalise({ services: { apps: multi } }); // valid inventory; the drill itself refuses (below)
}),
}),
+ test('objects: the declared object directory is archived, restored into scratch, and every blob sha256 checked', async () => {
+ const a = 'tenant page a\n';
+ const b = 'tenant asset b\n';
+ const pa = blob(OBJECTS_DIR, a);
+ const pb = blob(OBJECTS_DIR, b);
+ const host = await drillScenario({
+ objects: [{ name: 'objects', path: OBJECTS_DIR }],
+ blobs: [{ path: pa.path, content: a }, { path: pb.path, content: b }],
+ drill: {
+ env: { PORT: '{port}', HOST: '127.0.0.1', DISCORD_RELAY_ENABLED: '0', EVENTS_URL: '', UPLOAD_DIR: '{tmp}/data/uploads', GREETING: 'two words', HOST_STORAGE_DIR: '{tmp}/objects' },
+ dirs: ['{tmp}/data', '{tmp}/objects'],
+ },
+ });
+ const archive = `${host.backupDir}/objects.tar.gz`;
+ assert.deepStrictEqual([host.files.get(archive).owner, host.files.get(archive).mode], ['root', 0o600], 'the archive is an ordinary root-only copy');
+ const r = await host.cli('drill', 'tools', '--keep');
+ assert.strictEqual(r.code, 0, r.out);
+ const rec = lastLog(host);
+ assert.strictEqual(rec.kept, true);
+ assert.deepStrictEqual(rec.objects.map((o) => [o.name, o.blobs, o.skipped, o.integrity]), [['objects', 2, 0, 'sha256 ok']]);
+ const tmp = rec.dir;
+ // The archive is listed first, then extracted without restoring production's ownership or modes.
+ assert.ok(host.calls.some((c) => c.cmd === 'tar' && c.args[0] === '-tvzf' && c.args[1] === archive), 'listed before extraction');
+ const extract = host.calls.find((c) => c.cmd === 'tar' && c.args[0] === '-xzf');
+ assert.deepStrictEqual(extract.args, ['-xzf', archive, '-C', `${tmp}/objects`, '--no-same-owner', '--no-same-permissions']);
+ assert.strictEqual(extract.privileged, true);
+ // Restored into the drill's own scratch directory, never the live object directory.
+ assert.strictEqual(rec.objects[0].copy, `${tmp}/objects`);
+ assert.ok(rec.objects[0].copy.startsWith(`${tmp}/`));
+ assert.strictEqual(host.read(`${tmp}/objects/projects/prj_1/${pa.sha.slice(0, 2)}/${pa.sha}`), a);
+ assert.strictEqual(host.read(`${tmp}/objects/projects/prj_1/${pb.sha.slice(0, 2)}/${pb.sha}`), b);
+ // The instance is pointed at the restored directory, not production's.
+ assert.ok(host.read(`${tmp}/drill.env`).split('\n').includes(`HOST_STORAGE_DIR=${tmp}/objects`));
+ assert.match(rec.markdown, /object store `objects` 2 blob\(s\) sha256 ok/);
+ // The live objects were never written or moved.
+ assert.strictEqual(host.read(pa.path), a);
+ assert.strictEqual(host.read(pb.path), b);
+ }),
+
+ test('objects: a blob whose bytes do not match the sha256 in its name fails the drill before anything starts', async () => {
+ const good = 'good blob\n';
+ const p = blob(OBJECTS_DIR, good);
+ const wrongName = 'a'.repeat(64);
+ const host = await drillScenario({
+ objects: [{ name: 'objects', path: OBJECTS_DIR }],
+ blobs: [{ path: p.path, content: good }, { path: `${OBJECTS_DIR}/projects/prj_1/aa/${wrongName}`, content: 'not its name\n' }],
+ });
+ const before = fsSnapshot(host);
+ const r = await host.cli('drill', 'tools');
+ assert.strictEqual(r.code, 2, r.out);
+ const rec = lastLog(host);
+ assert.strictEqual(rec.failure.stage, 'objects');
+ assert.match(rec.failure.message, /blob\(s\) do not match their name/);
+ assert.match(rec.failure.message, /aa\/a{64}/);
+ assert.strictEqual(host.systemdRuns.length, 0, 'nothing is started');
+ assert.ok(!host.files.has(rec.dir));
+ assert.strictEqual(host.read(p.path), good, 'the live object is untouched');
+ for (const changed of changedPaths(before, fsSnapshot(host))) assert.ok(allowedWrite(changed, rec.dir), `unexpected write: ${changed}`);
+ }),
+
+ test('objects: a service declaring an object directory refuses to drill from a backup without its archive', async () => {
+ // The directory does not exist, so the backup records the object directory as skipped.
+ const host = await drillScenario({ objects: [{ name: 'objects', path: OBJECTS_DIR }], blobs: [] });
+ const r = await host.cli('drill', 'tools');
+ assert.strictEqual(r.code, 1);
+ assert.match(r.out, /has no copy of the objects object directory \(objects\.tar\.gz\)/);
+ assert.strictEqual(host.systemdRuns.length, 0);
+ }),
+
+ test('objects: a symlink or special-file member is refused before root extracts the archive', async () => {
+ for (const member of [{ type: 'l', name: './evil', target: '/' }, { type: 'p', name: './fifo' }]) {
+ const a = 'tenant page a\n';
+ const pa = blob(OBJECTS_DIR, a);
+ const host = await drillScenario({ objects: [{ name: 'objects', path: OBJECTS_DIR }], blobs: [{ path: pa.path, content: a }] });
+ host.tarMembers = [member];
+ const r = await host.cli('drill', 'tools');
+ assert.strictEqual(r.code, 2, `${member.type}: ${r.out}`);
+ const rec = lastLog(host);
+ assert.strictEqual(rec.failure.stage, 'objects');
+ assert.match(rec.failure.message, /not a regular file or directory/);
+ assert.ok(!host.calls.some((c) => c.cmd === 'tar' && c.args[0] === '-xzf'), `${member.type}: never extracted`);
+ assert.ok(!host.files.has(rec.dir), 'cleaned up');
+ }
+ }),
+
+ test('objects: a symlinked directory already in the restored tree is refused, not walked as root', async () => {
+ const host = scenario();
+ const dir = '/var/lib/openvibe-drills/tools-20260922-120100/objects';
+ const good = 'a good blob\n';
+ const sha = crypto.createHash('sha256').update(good).digest('hex');
+ host.put(`${dir}/projects/prj_1/${sha.slice(0, 2)}/${sha}`, good, { owner: 'ubuntu' });
+ // A file whose name IS a sha256 but whose bytes are not, behind a symlinked directory: if the
+ // walk followed the link it would hash this and report a mismatch.
+ const outsideSha = 'b'.repeat(64);
+ host.put(`/var/lib/openvibe-secret/${outsideSha}`, 'not its name\n', { owner: 'root', mode: 0o600 });
+ await host.exec.symlink('/var/lib/openvibe-secret', `${dir}/evil`);
+ const entry = { blobs: 0, skipped: 0, bytes: 0, mismatches: 0 };
+ await assert.rejects(
+ verifyObjectBlobs(host.exec, dir, entry),
+ (err) => /evil \(symlink or special file\)/.test(err.message) && !err.message.includes(outsideSha) && !err.message.includes('/var/lib/openvibe-secret'),
+ );
+ assert.strictEqual(entry.blobs, 1, 'the real blob was still checked');
+ assert.strictEqual(entry.mismatches, 1);
+ }),
+
+ test('objects: a drill with no room to extract the archive fails before root restores it', async () => {
+ const a = 'tenant page a\n';
+ const pa = blob(OBJECTS_DIR, a);
+ const host = await drillScenario({ objects: [{ name: 'objects', path: OBJECTS_DIR }], blobs: [{ path: pa.path, content: a }] });
+ const callsBefore = host.calls.length;
+ host.statfsFree = 1024;
+ const r = await host.cli('drill', 'tools');
+ assert.strictEqual(r.code, 2, r.out);
+ const rec = lastLog(host);
+ assert.strictEqual(rec.failure.stage, 'space');
+ assert.match(rec.failure.message, /not enough free space under \/var\/lib\/openvibe-drills/);
+ assert.ok(!host.calls.slice(callsBefore).some((c) => c.cmd === 'tar'), 'nothing was tarred or extracted');
+ assert.ok(!host.files.has(rec.dir));
+ }),
+
+ test('inventory: object directories are validated', () => {
+ const base = { owner: 'ubuntu', repo: '/opt/openvibe.host', units: [], databases: [{ name: 'host', engine: 'postgresql', database: 'ov_host' }] };
+ const svc = (objects) => normalise({ services: { h: { ...base, objects } } });
+ assert.deepStrictEqual(svc([{ name: 'objects', path: '/var/lib/openvibe-host-api/objects' }]).services.h.objects, [{ name: 'objects', path: '/var/lib/openvibe-host-api/objects' }]);
+ assert.deepStrictEqual(svc(undefined).services.h.objects, []);
+ assert.throws(() => svc([{ name: 'a/b', path: '/x' }]), /name must be a plain name/);
+ assert.throws(() => svc([{ name: 'objects', path: 'relative' }]), /absolute path/);
+ assert.throws(() => svc([{ name: 'objects', path: '/x' }, { name: 'objects', path: '/y' }]), /listed twice/);
+ assert.throws(() => svc('nope'), /must be a list of directories/);
+ }),
+
test('several units and neither drill.unit nor drill.command: refused', async () => {
test('several units and neither drill.unit nor drill.command: refused', async () => {
const host = await appsScenario({ drill: { unit: null } });
const host = await appsScenario({ drill: { unit: null } });
const r = await host.cli('drill', 'apps');
const r = await host.cli('drill', 'apps');
@@ -13,6 +13,7 @@
@@ -13,6 +13,7 @@
const crypto = require('crypto');
const crypto = require('crypto');
const path = require('path');
const path = require('path');
const { Readable, Writable } = require('stream');
const { Readable, Writable } = require('stream');
+const { createTarGz, readTarGz } = require('../lib/tar');
function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', start = Date.parse('2026-09-22T12:00:00Z') } = {}) {
function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', start = Date.parse('2026-09-22T12:00:00Z') } = {}) {
const files = new Map(); // path -> { type: 'file'|'dir'|'symlink', content, mode, owner, target }
const files = new Map(); // path -> { type: 'file'|'dir'|'symlink', content, mode, owner, target }
@@ -53,6 +54,8 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', st
@@ -53,6 +54,8 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', st
statfsFree: 100 * 1024 ** 3,
statfsFree: 100 * 1024 ** 3,
statfsSize: 500 * 1024 ** 3,
statfsSize: 500 * 1024 ** 3,
reads: [],
reads: [],
+ onTar: null, // (args, opts, real) -> undefined | { code, stdout, stderr } ; override a tar result
+ tarMembers: [], // [{ type, name, target }] extra (non-)members a `tar -tvzf` listing reports
certbot: null, // (args, opts) -> { code, stdout, stderr } | Error ; default: nothing to do
certbot: null, // (args, opts) -> { code, stdout, stderr } | Error ; default: nothing to do
certbotCalls: [], // { args, as, privileged }
certbotCalls: [], // { args, as, privileged }
journalctl: null, // (args, opts) -> { code, stdout, stderr } ; default: no entries
journalctl: null, // (args, opts) -> { code, stdout, stderr } ; default: no entries
@@ -365,6 +368,66 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', st
@@ -365,6 +368,66 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', st
return { code: 0, stdout: 'Done', stderr: '' };
return { code: 0, stdout: 'Done', stderr: '' };
}
}
+ // ── tar (backup/drill of a content-addressed object directory) ──
+ // The forms the backup and drill use: `tar -czf <dest> -C <dir> .`, `tar -tvzf <archive>` (the
+ // drill's member pre-check) and `tar -xzf <archive> -C <dir>`. The archive is a real gzipped tar
+ // (lib/tar.js), so it survives the off-host encrypt/decrypt pipeline byte for byte.
+ function tarCmd(args, opts) {
+ const result = tarCmdReal(args, opts);
+ // A test can override the result (to simulate a warning or a failure) after the real archive
+ // was made, so --exclude and the archive's contents stay realistic.
+ if (host.onTar) { const r = host.onTar(args, opts, result); if (r) return { stdout: '', stderr: '', ...r }; }
+ return result;
+ }
+ function tarCmdReal(args, opts) {
+ const mode = String(args[0] || '');
+ const archive = args[1];
+ const ci = args.indexOf('-C');
+ const dir = ci >= 0 ? args[ci + 1] : '.';
+ // `--exclude=PAT` (used for Host's transient <root>/tmp scratch): skip a member whose relative
+ // name is the pattern or sits under it.
+ const excludes = args.filter((a) => a.startsWith('--exclude=')).map((a) => a.slice(10).replace(/^\.\//, ''));
+ const excluded = (name) => excludes.some((ex) => name === ex || name.startsWith(`${ex}/`));
+ if (mode.includes('c')) {
+ const target = args[args.length - 1];
+ const base = resolveLinks(dir);
+ const root = resolveLinks(path.join(dir, target));
+ const entries = [];
+ for (const [k, v] of files) {
+ if (v.type !== 'file') continue;
+ if (k !== root && !k.startsWith(`${root}/`)) continue;
+ const name = path.relative(base, k);
+ if (excluded(name)) continue;
+ entries.push({ name, content: v.content });
+ }
+ put(archive, createTarGz(entries, clock), { owner: opts.as || user, mode: 0o600 });
+ return { code: 0, stdout: '', stderr: '' };
+ }
+ if (mode.includes('t')) {
+ const e = get(archive);
+ if (!e || e.type !== 'file') return { code: 1, stdout: '', stderr: `tar: ${archive}: No such file or directory` };
+ let parsed;
+ try { parsed = readTarGz(Buffer.isBuffer(e.content) ? e.content : Buffer.from(String(e.content))); } catch (err) { return { code: 1, stdout: '', stderr: `tar: ${err.message}` }; }
+ const lines = parsed.map((f) => `-rw-r--r-- ubuntu/ubuntu ${Buffer.byteLength(f.content)} 2026-09-22 12:00 ${f.name}`);
+ for (const m of host.tarMembers) lines.push(`${m.type}rw-r--r-- root/root 0 2026-09-22 12:00 ${m.name}${m.target ? ` -> ${m.target}` : ''}`);
+ return { code: 0, stdout: lines.length ? `${lines.join('\n')}\n` : '', stderr: '' };
+ }
+ if (mode.includes('x')) {
+ const e = get(archive);
+ if (!e || e.type !== 'file') return { code: 1, stdout: '', stderr: `tar: ${archive}: No such file or directory` };
+ const destdir = resolveLinks(dir);
+ let parsed;
+ try { parsed = readTarGz(Buffer.isBuffer(e.content) ? e.content : Buffer.from(String(e.content))); } catch (err) { return { code: 1, stdout: '', stderr: `tar: ${err.message}` }; }
+ for (const f of parsed) {
+ const name = f.name.replace(/^\.\//, '');
+ if (!name || name.split('/').includes('..')) continue;
+ put(path.join(destdir, name), f.content, { owner: opts.as || 'root', mode: 0o644 });
+ }
+ return { code: 0, stdout: '', stderr: '' };
+ }
+ return { code: 1, stdout: '', stderr: `fake tar: unsupported ${args.join(' ')}` };
+ }
+
// ── systemd ──
// ── systemd ──
host.addUnit = (unit, state = {}) => {
host.addUnit = (unit, state = {}) => {
units.set(unit, { load: 'loaded', active: 'active', sub: 'running', mainPid: 1000 + units.size, fragmentPath: `/etc/systemd/system/${unit}`, dropIns: [], partOf: [], restarts: 0, runningSha: null, ...state });
units.set(unit, { load: 'loaded', active: 'active', sub: 'running', mainPid: 1000 + units.size, fragmentPath: `/etc/systemd/system/${unit}`, dropIns: [], partOf: [], restarts: 0, runningSha: null, ...state });
@@ -499,6 +562,17 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', st
@@ -499,6 +562,17 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', st
const r = host.onNode ? host.onNode(args, opts) : undefined;
const r = host.onNode ? host.onNode(args, opts) : undefined;
return r ? { stdout: '', stderr: '', ...r } : { code: 0, stdout: '', stderr: '' };
return r ? { stdout: '', stderr: '', ...r } : { code: 0, stdout: '', stderr: '' };
}
}
+ case 'du': {
+ const target = args[args.length - 1];
+ const root = resolveLinks(target);
+ if (!files.has(root)) return { code: 1, stdout: '', stderr: `du: cannot access '${target}': No such file or directory` };
+ let total = 0;
+ for (const [k, v] of files) {
+ if (k !== root && !k.startsWith(`${root}/`)) continue;
+ total += v.type === 'file' ? Buffer.byteLength(Buffer.isBuffer(v.content) ? v.content : String(v.content)) : 4096;
+ }
+ return { code: 0, stdout: `${total}\t${target}\n`, stderr: '' };
+ }
case 'certbot': {
case 'certbot': {
host.certbotCalls.push({ args, as: opts.as || null, privileged: !!opts.privileged });
host.certbotCalls.push({ args, as: opts.as || null, privileged: !!opts.privileged });
const r = host.certbot ? host.certbot(args, opts) : undefined;
const r = host.certbot ? host.certbot(args, opts) : undefined;
@@ -572,6 +646,7 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', st
@@ -572,6 +646,7 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', st
return { code: 0, stdout: `${JSON.stringify(stanzas)}\n`, stderr: '' };
return { code: 0, stdout: `${JSON.stringify(stanzas)}\n`, stderr: '' };
}
}
case 'systemd-run': return systemdRunCmd(args);
case 'systemd-run': return systemdRunCmd(args);
+ case 'tar': return tarCmd(args, opts);
case 'cp': {
case 'cp': {
const src = args[args.length - 2];
const src = args[args.length - 2];
const dest = args[args.length - 1];
const dest = args[args.length - 1];