openvibers/openvibe.host Commits

Back up declared tenant object directories with the databases, offsite included, and verify every blob's sha256 in a restore drill (#23)

OP OpenVibers committed 5 Oct 2026 · 1a1f1f71e8 · 12 files, +603 −41

* Back up declared tenant object directories with the databases, offsite included, and verify every blob's sha256 in a restore drill

## Changes (file:line)

- `lib/inventory.js:152` — `normaliseObjects()`: per-service `objects` list of `{name, path}`, name a plain name (unique), path absolute; `lib/inventory.js:261` — normalised `objects` field on every service.
- `lib/commands/backup.js:135,163,322,221` — treat a service with only objects as backup-able; don't early-return verification-only; in `backupAll` don't skip it; after the database loop, archive each declared directory as `<name>.tar.gz` (`tar -czf <stage>/<name>.tar.gz -C <path> .`), root-created and then locked to root:0600/validated by the existing plain-file loop, recorded with `kind: 'objects'`.
- `lib/offsite.js:255` — manifest entries carry `kind` (encryption/HMAC pipeline unchanged; `push` already uploads any `dest`); `lib/offsite.js:335` — `unsafeOut()` also protects live object directories; `lib/offsite.js:403` — `restore-download` treats an object archive as an opaque artifact (manifest sha256 already verified) instead of SQLite-quick-checking it.
- `lib/commands/drill.js:156,185` — `BLOB_RE` and `verifyObjectBlobs()` (recursive walk, streamed sha256 per 64-hex-named file, refuses a blob-named non-plain file, skips non-blob names like Host's `tmp/` scratch); `:240` — `resolveBackup` requires the `<name>.tar.gz` archive; `:452` — extract into the scratch dir `<tmp>/<name>` and verify; `:350,669` — record and Markdown row.
- `test/fake-host.js:373,614` — fake `tar` create/extract (`-czf … -C dir .`, `-xzf … -C dir`) using the real gzipped tar from `lib/tar.js`, so archives survive the encrypt/decrypt pipeline.
- `host.example.json:979` — `host` declares `objects: [{name:"objects", path:"/var/lib/openvibe-host-api/objects"}]`; notes at `:943`/`:1039` updated.
- `README.md:407,411`; `docs/threat-review.md:288`; `docs/backups.md:37`; `docs/restore-drills.md:95,190` — the "objects not backed up" records and the inventory/drill docs updated.
- `test/drill.test.js:560,594,615,624` — declared-object drill (restores into scratch, checks 2 blobs, live dir untouched, instance env points at it), sha mismatch fails at stage `objects`, missing archive refused, `objects` validation; `test/backup-offsite.test.js:88`+ — `--offsite` archives/encrypts/uploads the tree and records `kind`, and `restore-download` handling.

## Where origin/main differed

- README's "objects not backed up" line is `411`, not 407 (407 is the "no backup beyond `ovhost backup host`" bullet); both updated. `docs/threat-review.md` row is `288`, not 285.
- `lib/offsite.js` manifest loop is 240–258; it already uploads any file with a `dest`, so the only change needed there was `kind` plus the `restore-download`/`unsafeOut` handling — the encryption and HMAC path is untouched.
- `backup()` already had no early-exit for a service with objects, but its "no databases" guard and verification-only return did need widening.

## Tests

`ov test test/backup-offsite.test.js test/drill.test.js` → **2 passed**. I also ran the two example-inventory-reading files my `host.example.json` edit reaches: `ov test test/backup-offsite.test.js test/drill.test.js test/cli-inventory.test.js test/drill-live-media.test.js` → **4 passed**. No commit or push made.

* Object backups: root-locked stage before tarring, symlink and special members refused on restore, no owner or mode restore, busy-store warnings tolerated, space checked before tar and extract (review)

All five findings fixed and verified.

**1. Root wrote the object archive into a service-owned stage** — `lockDown(exec, stage, 0o700)` moved ahead of the object loop, and any pre-planted entry at the archive path is removed before root's tar opens it; tests assert root chowns/chmods the stage before tar (`lib/commands/backup.js:230`, `:260`; test `backup-offsite.test.js` "written only after root has taken the staging directory over").

**2. `walkFiles` followed symlinked directories as root** — recursion now requires `st.isDir && !st.isSymlink`, and `verifyObjectBlobs` refuses every symlink or special file regardless of name (`lib/commands/drill.js:168`, `:195`; unit test "a symlinked directory already in the restored tree is refused, not walked as root").

**3. Root `tar -xzf` restored links/special files with ownership and modes** — the drill now pre-lists with `tar -tvzf` (rejecting non-regular/non-directory, absolute and `..` members, plus tar's "Removing leading" stderr) and extracts with `--no-same-owner --no-same-permissions` (`lib/commands/drill.js:212`, `:497`, `:502`; tests assert the flags in the objects test and that symlink/fifo members are refused before extraction).

**4. A busy store failed the backup** — Host's scratch `./tmp` is excluded (`--exclude=./tmp`, `--exclude=./tmp/*`) and a tar exit of 1 whose stderr is only file-changed/removed warnings is accepted via `tarWarningsOnly` (`lib/commands/backup.js:262-263`; tests cover warning-accepted-with-scratch-excluded and a genuine failure still failing).

**5. No staging/extraction space check** — backup records `du -sb` of the source and checks free space under `backupStagingDir` before tarring (`lib/commands/backup.js:245-253`); the record carries `sourceBytes` into the drill, which checks free space under `drillDir` before extracting (`lib/commands/drill.js:274`, `:419`; tests for both no-room paths).

Test results: `ov test test/drill.test.js test/backup-offsite.test.js test/cli-inventory.test.js` → **3 passed** (all 26 drill tests and 26 backup/offsite tests run individually, including the 7 new ones). No commit or push made.

Changes

ddiff --git a/README.md b/README.md
iindex 908e06b..13fa2de 100644
---- a/README.md
++++ b/README.md
@@@ -404,11 +404,11 @@ The service runs on the host (loopback, since 2026-09-23) and the pending tenant
### Not done yet (Stage B)
- Uploads are held in memory while they are validated (bounded by `HOST_MAX_UPLOAD_BYTES` and `HOST_MAX_UNPACKED_BYTES`), so the service needs that much headroom per concurrent upload.
--- Objects live on the service's local disk, not in OpenVibe.Media; there is no replication and no backup beyond `ovhost backup host`. The database (`ov_host` on the host's data role, ADR-035; schema in [migrations/](migrations/)) is backed up with the others by pgBackRest.
++- Objects live on the service's local disk, not in OpenVibe.Media; there is no replication. `ovhost backup host` archives `/var/lib/openvibe-host-api/objects` (the inventory's `objects` entry) alongside `ov_host`, uploads the encrypted archive off-host, and a restore drill extracts it and checks every blob's sha256. The database (`ov_host` on the host's data role, ADR-035; schema in [migrations/](migrations/)) is backed up with the others by pgBackRest.
- Certificates for custom domains are issued and installed by `ovhost certs renew --install` (step 8) on the `openvibe-certs.timer`; certbot stays outside the Host service and no key file is read.
- Projects are created in Host. When Network has projects (ADR-014), Host should accept only Network project ids and read membership from Network.
- `openvibe.host` is not on the Public Suffix List, so a tenant page can still set `Domain=openvibe.host` cookies (a cookie bomb breaks the dashboard and other tenant sites for that visitor). The protections above do not depend on it. [docs/threat-review.md §5](docs/threat-review.md#5-the-public-suffix-list-question) records the decision (launch without it) and the recommended follow-up (dashboard off the tenant zone, then list the zone).
--- Tenant objects are not in `ovhost backup` (only `host.db`).
++- Tenant objects are in `ovhost backup`: the inventory declares them as `objects` (`/var/lib/openvibe-host-api/objects`), the backup archives the tree, the off-host copy encrypts it, and a drill extracts it and checks every blob's sha256.
- Git deploys are Phase 1 only (see [Git deploys](#git-deploys-phase-1)): the project's own CI builds and posts the output. Host never clones, fetches, holds a repository credential or builds; in-Host builds belong to Run (plan T14). There is no webhook: a push reaches Host only through the CI's call.
- Tenant sites publish their own sitemap/robots/feed; when a deploy ships no `sitemap.xml` or `robots.txt`, Host generates one (the manifest's HTML pages on the host the request came in on, so a verified custom domain gets its own `sitemap.xml`; sandbox projects get `Disallow: /` and are `noindex`). A tenant's own file at either path always wins. The dashboard host serves `/robots.txt` and `/sitemap.xml` (front page and legal pages).
- The Codes portal (Wave 20) does not exist, so there is no public developer onboarding for Host yet.
ddiff --git a/docs/backups.md b/docs/backups.md
iindex f19527e..ca31a0b 100644
---- a/docs/backups.md
++++ b/docs/backups.md
@@@ -14,7 +14,7 @@ Restore drills (a second instance started from a backup and compared with produc
## What is backed up
`ovhost backup --all` backs up every service in the inventory (`/etc/openvibe/host.json`) that has
--a `databases` list. `databases[]` items are one of two **engines** ([db-inventory.md](db-inventory.md)):
++a `databases` or `objects` list. `databases[]` items are one of two **engines** ([db-inventory.md](db-inventory.md)):
SQLite, a file the service user's worker copies; and PostgreSQL, a database in the host's cluster
(ADR-035) that ovhost verifies through pgBackRest and periodically dumps. With `host.example.json`:
@@@ -34,6 +34,13 @@ services](#postgresql-services) covers what that means.
A database that does not exist yet (a service that is not deployed) is reported as `skipped` and
does not fail the run. Adding a database to the inventory is all it takes to include it.
++**Object directories.** A service may also declare `objects`: a list of `{ name, path }` directories
++whose files are content-addressed, each named by its own sha256. OpenVibe.Host's tenant object store
++(`HOST_STORAGE_DIR=/var/lib/openvibe-host-api/objects`) is the first. `ovhost backup` archives each
++directory as `<name>.tar.gz` (root:root 0600, like every other copy); the off-host code encrypts and
++uploads it with the rest, and a [drill](restore-drills.md) extracts it into its own scratch directory
++and checks every blob's sha256.
++
## One run
For each service, one after another:
@@@ -448,7 +455,7 @@ Do this at least once after installing, and again after any change to the key or
| Command | What it does | Exit |
|---|---|---|
--| `ovhost backup --all [--offsite] [--logical] [--no-prune] [--keep-daily n] [--keep-weekly n]` | back up every service with databases (verify pgBackRest, dump SQLite, dump PostgreSQL when due or `--logical`); prune; summary; optionally upload | 0 all ok · 1 lock/usage · 2 any failure |
++| `ovhost backup --all [--offsite] [--logical] [--no-prune] [--keep-daily n] [--keep-weekly n]` | back up every service with databases or object directories (verify pgBackRest, dump SQLite, dump PostgreSQL when due or `--logical`, archive object directories); prune; summary; optionally upload | 0 all ok · 1 lock/usage · 2 any failure |
| `ovhost offsite push [--run <run>]` | upload a run (default: the latest) again, for example after a failed upload | 0 · 1 config · 2 failure |
| `ovhost offsite list [<service>] [--from-host <h>]` | runs off-host, newest first | 0 · 1 config · 2 bucket error |
| `ovhost offsite check` | config, key and bucket access; uploads nothing | 0 · 1 config · 2 bucket error |
ddiff --git a/docs/restore-drills.md b/docs/restore-drills.md
iindex 9b223f8..66e2165 100644
---- a/docs/restore-drills.md
++++ b/docs/restore-drills.md
@@@ -92,6 +92,9 @@ block (see `host.example.json`):
`PRAGMA integrity_check`, run as the service user, must return `ok` for each copy;
otherwise the drill fails before anything starts. A PostgreSQL service restores its `.dump` into a
scratch database instead — see [A PostgreSQL drill](#a-postgresql-drill).
++ Every `objects` directory the service declares is restored as well: its archive is extracted into
++ `<tmp>/<name>`, owned by the service user and never the live directory, and every blob is checked
++ against the sha256 in its file name.
2. **Start.** A second instance starts from the production checkout through `systemd-run`, as the
service user. It uses the production unit's `ExecStart` and `WorkingDirectory`, the production
unit's non-secret `Environment=` values and the production env file. After that it loads
@@@ -184,7 +187,7 @@ Supported with declared overrides (the table above records which have passed on
- network
- community
- events
--- host (the Host API: `host.db` only, counts only)
++- host (the Host API: the `ov_host` database and the tenant object store, whose blobs are sha256-checked; counts only)
- billing
- chat
- search
ddiff --git a/docs/threat-review.md b/docs/threat-review.md
iindex d07378b..e9d23e0 100644
---- a/docs/threat-review.md
++++ b/docs/threat-review.md
@@@ -285,6 +285,6 @@ the history of any site.
| Unverified or lapsed custom domains reach nginx's default server | medium (confusion, mis-served content on a stranger's domain) | operator: catch-all `default_server` (docs/launch.md step 0) |
| Cookie bomb across tenants and the dashboard (no PSL) | low (per-visitor nuisance; takedown exists) | owner decision (§5) |
| No abuse mailbox/owner | medium (process, not code) | owner |
--| Objects not backed up | medium for tenants (alpha, documented) | follow-up: add the object store to the backup run |
++| Objects not backed up | medium for tenants (alpha, documented) | fixed: the inventory's `objects` entry archives the store, the off-host copy encrypts it, and a drill restores it and checks every blob's sha256 |
| Decompressing an upload blocked the event loop (`gunzipSync`) | done: async `zlib.gunzip` on the libuv threadpool (`server/artifacts/archive.js`), same `maxOutputLength` cap | done |
| Takedown routes not in released `openvibe-contracts` `host.site.manage.implementedBy` | none at runtime (the contract check passes) | Contracts: next release (docs/launch.md) |
ddiff --git a/host.example.json b/host.example.json
iindex 388e141..f4cf4c2 100644
---- a/host.example.json
++++ b/host.example.json
@@@ -940,7 +940,7 @@
}
},
"host": {
-- "_note": "OpenVibe.Host Stage B (tenant static hosting): the Host API service on port 4910. Not deployed as of 22 Sep 2026. Its content-addressed objects live in /var/lib/openvibe-host-api, which is NOT ovhost's own stateDir (/var/lib/openvibe-host); its database is the PostgreSQL database ov_host. Tenant vhosts: `ovhost nginx tenants host`.",
++ "_note": "OpenVibe.Host Stage B (tenant static hosting): the Host API service on port 4910. Not deployed as of 22 Sep 2026. Its content-addressed objects live in /var/lib/openvibe-host-api/objects (`HOST_STORAGE_DIR`), which is NOT ovhost's own stateDir (/var/lib/openvibe-host); its database is the PostgreSQL database ov_host. Tenant vhosts: `ovhost nginx tenants host`.",
"repo": "/opt/openvibe.host",
"units": [
"openvibe-host.service"
@@@ -976,6 +976,12 @@
"database": "ov_host"
}
],
++ "objects": [
++ {
++ "name": "objects",
++ "path": "/var/lib/openvibe-host-api/objects"
++ }
++ ],
"nginx": {
"tenants": {
"sitesDomain": "openvibe.host",
@@@ -1036,7 +1042,7 @@
"table": "host_domains"
}
],
-- "_notes": "Restores the ov_host database only: `ovhost backup host` does not copy the content-addressed objects, so tenant content cannot be compared (every /api/v1 read needs auth). HOST_WORKER=off stops DNS rechecks and the object sweep."
++ "_notes": "Restores the ov_host database and the archived `objects` directory into {tmp}/objects (HOST_STORAGE_DIR), checking every blob's sha256; tenant content is still not compared (every /api/v1 read needs auth). HOST_WORKER=off stops DNS rechecks and the object sweep."
}
},
"billing": {
ddiff --git a/lib/commands/backup.js b/lib/commands/backup.js
iindex be56a15..d629799 100644
---- a/lib/commands/backup.js
++++ b/lib/commands/backup.js
@@@ -1,7 +1,8 @@
'use strict';
/**
* ovhost backup <service>: an online copy of every database the inventory declares for the service,
-- * into <backupDir>/<service>/<YYYYMMDD-HHMMSS>/<name>.<ext>, recorded in
++ * and a tar archive of every content-addressed object directory it declares, into
++ * <backupDir>/<service>/<YYYYMMDD-HHMMSS>/<name>.<ext>, recorded in
* <stateDir>/backups/<service>.jsonl. SQLite is copied by a worker running as the service user (root
* never opens a service database — see sqlite-worker.js) and checked with PRAGMA quick_check.
* PostgreSQL is not dumped every night: pgBackRest holds the cluster's full + differential backups
@@@ -30,10 +31,11 @@
* throws a BackupError carrying the record. A SQLite backup that found none of its databases is
* recorded as ok: false, empty: true and leaves no directory behind; PostgreSQL never sets empty.
*
-- * ovhost backup --all (backupAll): every service that declares databases, one after another. A
-- * failing service never stops the others. Afterwards each service's backups are pruned to the last
-- * 7 daily and 4 weekly good ones (lib/retention.js), and a JSON summary of the run is written to
-- * <stateDir>/backup-runs/<run>.json. With --offsite the run is then uploaded (lib/offsite.js).
++ * ovhost backup --all (backupAll): every service that declares databases or object directories, one
++ * after another. A failing service never stops the others. Afterwards each service's backups are
++ * pruned to the last 7 daily and 4 weekly good ones (lib/retention.js), and a JSON summary of the run
++ * is written to <stateDir>/backup-runs/<run>.json. With --offsite the run is then uploaded
++ * (lib/offsite.js).
*/
const path = require('path');
const { service } = require('../inventory');
@@@ -46,6 +48,19 @@ const DAY_MS = 86400000;
// differentials: it exists for the off-site copy and per-service drills, not for durability.
const LOGICAL_DUMP_INTERVAL_DAYS = 7;
const ARCHIVER_STATE = 'pgbackrest-archiver.json';
++// Room a tar archive of an object directory needs beyond `du -sb` of its source: per-file tar headers
++// and a little gzip overhead for already-compressed blobs.
++const ARCHIVE_HEADROOM = 1024 * 1024;
++// Host stages an upload at <store root>/tmp/<random>.part and renames it into place, so the store is
++// never quiescent while it is tarred. These warnings are not failures: the scratch is excluded too.
++const BENIGN_TAR_RE = /file changed as we read it|file removed before we read it|Cannot stat: No such file or directory/i;
++
++/** GNU tar exits 1 when a file changed or vanished mid-read. That is not a failed archive when every
++ * warning is one of those; any other stderr line means the run failed. */
++function tarWarningsOnly(stderr) {
++ const lines = String(stderr || '').split('\n').map((l) => l.trim()).filter(Boolean);
++ return lines.length > 0 && lines.every((l) => BENIGN_TAR_RE.test(l));
++}
class BackupError extends Error {
constructor(message, record) { super(message); this.record = record; }
@@@ -130,7 +145,7 @@ async function lastGoodDumpAt(exec, inv, id) {
async function backup(ctx, id, { reason = 'manual', logical = false } = {}) {
const { exec, inv, log } = ctx;
const svc = service(inv, id);
-- if (!svc.databases.length) throw new Error(`${id} declares no databases`);
++ if (!svc.databases.length && !svc.objects.length) throw new Error(`${id} declares no databases or object directories`);
const postgres = svc.databases.filter((d) => d.engine === 'postgresql');
const hasPostgres = postgres.length > 0;
const engine = hasPostgres ? 'postgresql' : 'sqlite';
@@@ -156,8 +171,9 @@ async function backup(ctx, id, { reason = 'manual', logical = false } = {}) {
const at = last === null ? NaN : Date.parse(last);
if (logical || !Number.isFinite(at) || exec.now() - at > LOGICAL_DUMP_INTERVAL_DAYS * DAY_MS) dump.push(...postgres);
}
-- // A verification-only run (PostgreSQL, no dump due, no SQLite database) leaves nothing on disk.
-- if (!dump.length && !svc.databases.some((d) => d.engine === 'sqlite')) {
++ // A verification-only run (PostgreSQL, no dump due, no SQLite database, no objects) leaves
++ // nothing on disk.
++ if (!dump.length && !svc.databases.some((d) => d.engine === 'sqlite') && !svc.objects.length) {
const record = { service: id, at: new Date(exec.now()).toISOString(), reason, engine, ok: true, files: [], verified };
await exec.appendFile(path.join(inv.stateDir, 'backups', `${id}.jsonl`), `${JSON.stringify(record)}\n`);
return record;
@@@ -211,9 +227,46 @@ async function backup(ctx, id, { reason = 'manual', logical = false } = {}) {
}
files.push({ name: db.name, source: db.database, dest: path.join(dir, file), file, check: 'pg_dump -Fc', seconds: Math.round((exec.now() - t0) / 1000) });
}
-- // Take the staging directory back first: after this the service user (or postgres) can no
-- // longer reach anything inside it, so what is checked below cannot change underneath.
++ // Take the staging directory back FIRST: while the service user (or postgres) owns it, it could
++ // plant a symlink at the path root is about to write the object archive through, and root's
++ // `tar -czf` would follow it. Once root:root 0700, neither can reach it, and everything written
++ // below is checked before the move.
await lockDown(exec, stage, 0o700);
++ // Content-addressed object directories: one tar archive each, created by root AFTER the stage
++ // is root-locked. The drill extracts this archive into its own scratch directory and checks
++ // every blob's sha256; nothing here needs to open a blob individually.
++ for (const obj of svc.objects) {
++ const file = `${obj.name}.tar.gz`;
++ const archive = path.join(stage, file);
++ if (!(await exec.stat(obj.path))) { files.push({ name: obj.name, source: obj.path, skipped: 'not found' }); log(`skip ${obj.path}: not found`); continue; }
++ const t0 = exec.now();
++ // `du -sb` of the source: the uncompressed size the drill must find room for, and the
++ // figure the staging space check below is made against.
++ const du = await exec.run('du', ['-sb', '--', obj.path], { privileged: true });
++ const sourceBytes = du.code === 0 ? Number(String(du.stdout).trim().split(/\s+/)[0]) : null;
++ if (Number.isFinite(sourceBytes)) {
++ const free = (await exec.statfs(inv.backupStagingDir)).free;
++ const needed = sourceBytes + ARCHIVE_HEADROOM;
++ if (free < needed) {
++ files.push({ name: obj.name, source: obj.path, dest: path.join(dir, file), error: `not enough free space under ${inv.backupStagingDir}: ${free} bytes free, need ${needed} for ${obj.name}` });
++ log(`FAILED ${obj.path}: not enough free space under ${inv.backupStagingDir} (${free} < ${needed})`);
++ continue;
++ }
++ }
++ // Clear whatever might sit at the archive's path — a link planted while the service user
++ // owned the stage would survive lockDown — before root's tar opens it for writing.
++ await exec.run('rm', ['-rf', '--', archive], { privileged: true });
++ log(`backing up object directory ${obj.path} → ${path.join(dir, file)}`);
++ // Host's transient upload scratch lives at <root>/tmp/*.part; exclude it, and tolerate a
++ // tar exit of 1 whose only warnings are a file that changed or vanished mid-read.
++ const r = await exec.run('tar', ['-czf', archive, '-C', obj.path, '--exclude=./tmp', '--exclude=./tmp/*', '.'], { privileged: true });
++ if (r.code !== 0 && !(r.code === 1 && tarWarningsOnly(r.stderr))) {
++ files.push({ name: obj.name, source: obj.path, dest: path.join(dir, file), error: `tar: ${(r.stderr || '').trim().slice(0, 200)}` });
++ log(`FAILED ${obj.path}: ${(r.stderr || '').trim().slice(0, 200)}`);
++ continue;
++ }
++ files.push({ name: obj.name, source: obj.path, dest: path.join(dir, file), file, kind: 'objects', sourceBytes, check: 'object archive', seconds: Math.round((exec.now() - t0) / 1000) });
++ }
for (const f of files.filter((x) => x.file && !x.error)) {
const p = path.join(stage, f.file);
const fst = await exec.stat(p);
@@@ -299,7 +352,7 @@ async function backupAll(ctx, { daily = 7, weekly = 4, prune: doPrune = true, of
const file = path.join(inv.stateDir, 'backup-runs', `${run}.json`);
try {
for (const svc of Object.values(inv.services)) {
-- if (!svc.databases.length) continue;
++ if (!svc.databases.length && !svc.objects.length) continue;
const entry = { service: svc.id, status: 'ok', dir: null, files: [], error: null, pruned: [] };
summary.services.push(entry);
try {
ddiff --git a/lib/commands/drill.js b/lib/commands/drill.js
iindex 40bd9b2..4954a49 100644
---- a/lib/commands/drill.js
++++ b/lib/commands/drill.js
@@@ -12,6 +12,9 @@
* postgres, verify it with `pg_restore --list`, then create a scratch database and
* a login role `ov_<id>_drill_<stamp>` and `pg_restore --no-owner --role=…` the
* archive into the scratch database. Both are dropped again in a finally.
++ * Every object directory the service declares is also restored: its archive is extracted into
++ * <tmp>/<name> (owned by the service user, never the live directory) and every blob is checked
++ * against the sha256 in its file name.
* 2. Start a second instance from the production checkout, as the service user, through
* systemd-run. It uses the production unit's ExecStart and WorkingDirectory, the production
* env file, and a second env file with the drill's overrides: its port, the restored database
@@@ -43,6 +46,8 @@
* allows loopback).
*/
const path = require('path');
++const crypto = require('crypto');
++const { pipeline } = require('stream/promises');
const { service } = require('../inventory');
const envfile = require('../envfile');
const systemd = require('../systemd');
@@@ -147,12 +152,83 @@ async function countRows(exec, db, table, as) {
return n;
}
++// A content-addressed blob is named by its own sha256: that name is the whole integrity check.
++const BLOB_RE = /^[0-9a-f]{64}$/;
++
++/** Every non-directory entry under `dir`, recursively (regular files and anything else tar restored). */
++async function walkFiles(exec, dir) {
++ const out = [];
++ for (const e of (await exec.readdir(dir)) || []) {
++ const p = path.join(dir, e.name);
++ const st = await exec.stat(p);
++ if (!st) continue;
++ // Never follow a symlink: `exec.stat` follows it, so a restored member `x -> /` would otherwise
++ // be recursed into and walk (and hash) files outside the scratch directory as root. It is
++ // returned as a leaf so verifyObjectBlobs refuses it.
++ if (st.isDir && !st.isSymlink) out.push(...await walkFiles(exec, p));
++ else out.push(p);
++ }
++ return out;
++}
++
++/** The sha256 of a file, streamed so a large blob is never held in memory. */
++async function sha256Stream(exec, p) {
++ const hash = crypto.createHash('sha256');
++ await pipeline(exec.readStream(p), hash);
++ return hash.digest('hex');
++}
++
++/**
++ * Check a restored object directory: every regular file is a blob named by its own sha256. A file
++ * whose name is not a sha256 is not a blob (Host keeps transient write scratch under its store
++ * root) and is counted as skipped, never verified. Any symlink or special file (device, fifo,
++ * socket) in the restored tree is refused whatever its name, as is a blob-named entry that is not a
++ * plain file, as in a database copy. `entry` gets the counts and the sha256 mismatch list.
++ */
++async function verifyObjectBlobs(exec, dir, entry) {
++ const bad = [];
++ for (const p of await walkFiles(exec, dir)) {
++ const st = await exec.stat(p);
++ if (!st) continue;
++ const name = path.basename(p);
++ // walkFiles returns symlinks as leaves; a special file is neither a file nor a directory.
++ if (st.isSymlink || !st.isFile) { bad.push(`${path.relative(dir, p)} (symlink or special file)`); continue; }
++ if (!BLOB_RE.test(name)) { entry.skipped += 1; entry.bytes += st.size; continue; }
++ if (st.nlink !== 1) { bad.push(`${path.relative(dir, p)} (not a plain file)`); continue; }
++ entry.bytes += st.size;
++ const hash = await sha256Stream(exec, p);
++ if (hash !== name) { bad.push(`${path.relative(dir, p)} (sha256 ${hash})`); continue; }
++ entry.blobs += 1;
++ }
++ entry.mismatches = bad.length;
++ if (bad.length) throw new Error(`${bad.length} blob(s) do not match their name: ${bad.slice(0, 3).join(', ')}${bad.length > 3 ? ', …' : ''}`);
++}
++
++/**
++ * A `tar -tvzf` listing line: permissions, owner/group, size, date, time, name. Only a regular file
++ * ('-') or a directory ('d') may be extracted as root; a symlink, hard link or special file, or an
++ * absolute or `..` name, could write outside the scratch directory. -> the first offending member.
++ */
++function tarMemberProblem(listing) {
++ for (const line of String(listing || '').split('\n')) {
++ if (!line.trim()) continue;
++ const m = /^(\S+)\s+\S+\s+\d+\s+\S+\s+\S+\s+(.*)$/.exec(line);
++ if (!m) return `unparsable tar listing line: ${line.trim().slice(0, 80)}`;
++ const type = m[1][0];
++ const name = m[2].replace(/ -> .*$/, '').trim();
++ if (type !== '-' && type !== 'd') return `member ${name} is not a regular file or directory (${type})`;
++ if (name.startsWith('/') || name.split('/').includes('..')) return `member ${name} is not a safe relative path`;
++ }
++ return null;
++}
++
// ── inputs ───────────────────────────────────────────────────────────────────
/** The backup to restore: --backup <dir>, else the newest usable record in <stateDir>/backups/<id>.jsonl. */
async function resolveBackup(exec, inv, svc, dirArg) {
let dir = dirArg;
let takenAt = null;
++ let record = null;
const restored = Object.keys(svc.drill.databases).map((name) => svc.databases.find((d) => d.name === name));
const pgTargets = restored.filter((d) => d.engine === 'postgresql');
if (dir) {
@@@ -171,6 +247,7 @@ async function resolveBackup(exec, inv, svc, dirArg) {
const last = usable[usable.length - 1];
dir = last.dir;
takenAt = last.at || null;
++ record = last;
}
const st = await exec.stat(dir);
if (!st || !st.isDir) throw new DrillError(`backup directory ${dir} not found`);
@@@ -184,7 +261,20 @@ async function resolveBackup(exec, inv, svc, dirArg) {
if (db.engine === 'postgresql') files.push({ name, file, production: db, engine: 'postgresql', url: target.url, directUrl: target.directUrl });
else files.push({ name, file, production: db.path, envName: target.env, dir: target.dir, ext: path.extname(db.path) || '.db', engine: 'sqlite' });
}
-- return { dir, takenAt, files };
++ // Object directories are restored from the same backup: the archive holds the declared
++ // directory's contents, so the drill must not run without it (there would be nothing to verify).
++ const objects = [];
++ for (const obj of svc.objects) {
++ const file = path.join(dir, `${obj.name}.tar.gz`);
++ const fst = await exec.stat(file);
++ if (!fst || !fst.isFile) throw new DrillError(`backup ${dir} has no copy of the ${obj.name} object directory (${path.basename(file)})`);
++ // The uncompressed size recorded at backup time (`du -sb`), so the drill can check it has room
++ // to extract. A --backup directory has no record; its compressed size is the best we can do.
++ const rec = record && (record.files || []).find((f) => f.file === `${obj.name}.tar.gz`);
++ const sourceBytes = rec && Number.isFinite(rec.sourceBytes) ? rec.sourceBytes : fst.size;
++ objects.push({ name: obj.name, file, production: obj.path, sourceBytes });
++ }
++ return { dir, takenAt, files, objects };
}
/** How production starts the service: ExecStart, WorkingDirectory and non-secret Environment=. */
@@@ -287,6 +377,7 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
unit: unitName,
pid: null,
databases: [],
++ objects: [],
ready: null,
compare: [],
counts: [],
@@@ -322,6 +413,15 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
for (const name of left.databases) { log(`dropping a scratch database an earlier drill left: ${name}`); await dbengine.dropDatabase(exec, name); }
for (const name of left.roles) { log(`dropping a scratch role an earlier drill left: ${name}`); await dbengine.dropRole(exec, name); }
}
++ // Free space for the object archives too: extracting one writes its whole uncompressed
++ // contents under the drill directory. The size `du -sb` measured at backup time travels in the
++ // backup record; a --backup directory falls back to the archive's compressed size.
++ const objectBytes = bk.objects.reduce((n, o) => n + (o.sourceBytes || 0), 0);
++ if (objectBytes) {
++ await exec.mkdir(inv.drillDir, { mode: 0o711 });
++ const free = (await exec.statfs(inv.drillDir)).free;
++ if (free < objectBytes) throw fail('space', `not enough free space under ${inv.drillDir}: ${free} bytes free, need ${objectBytes} (the object directories' contents)`);
++ }
if (await exec.stat(tmp)) throw fail('prepare', `${tmp} already exists`);
await exec.mkdir(inv.drillDir, { mode: 0o711 });
// postgres reads the dump copy in <tmp>/pg, so it must be able to traverse <tmp> (0711: no listing;
@@@ -385,6 +485,29 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
if (entry.integrity !== 'ok') throw fail('integrity', `${f.name}: integrity_check = ${entry.integrity.slice(0, 300)}`);
log(`integrity_check ${f.name}: ok`);
}
++ // 1b. restore every declared object directory and check every blob's sha256. The archive is
++ // root-only, so root extracts it into the drill's own scratch directory — never over the live
++ // object directory, whose path reaches the drill instance only through its env override.
++ for (const obj of bk.objects) {
++ const dest = path.join(tmp, obj.name);
++ await exec.mkdir(dest, { owner: svc.runAs, mode: 0o700 });
++ // List the archive before extracting as root: only regular files and directories, with
++ // relative names, may be written. `--no-same-owner --no-same-permissions` then keeps the
++ // archive from restoring production's ownership or modes onto the drill copy.
++ const list = await exec.run('tar', ['-tvzf', obj.file], { privileged: true, env: { LC_ALL: 'C' } });
++ if (list.code !== 0) throw fail('objects', `listing ${obj.file} failed: ${(list.stderr || '').trim().slice(0, 200)}`);
++ if (/Removing leading|Member name contains '\.\.'/.test(list.stderr || '')) throw fail('objects', `${obj.file}: the archive holds an absolute or '..' member`);
++ const problem = tarMemberProblem(list.stdout);
++ if (problem) throw fail('objects', `${obj.file}: ${problem}`);
++ const x = await exec.run('tar', ['-xzf', obj.file, '-C', dest, '--no-same-owner', '--no-same-permissions'], { privileged: true });
++ if (x.code !== 0) throw fail('objects', `extracting ${obj.file} failed: ${(x.stderr || '').trim().slice(0, 200)}`);
++ const entry = { name: obj.name, source: obj.file, copy: dest, blobs: 0, skipped: 0, bytes: 0, mismatches: 0, integrity: null };
++ record.objects.push(entry);
++ log(`restoring object directory ${obj.file} → ${dest} (${svc.runAs})`);
++ try { await verifyObjectBlobs(exec, dest, entry); } catch (err) { entry.integrity = `error: ${err.message}`; throw fail('objects', `${obj.name}: ${err.message}`); }
++ entry.integrity = 'sha256 ok';
++ log(`object directory ${obj.name}: ${entry.blobs} blob(s) sha256 ok${entry.skipped ? `, ${entry.skipped} non-blob file(s) skipped` : ''}`);
++ }
const restoredCounts = [];
for (const c of d.counts) {
const rdb = svc.databases.find((x) => x.name === c.db);
@@@ -589,6 +712,9 @@ function markdownRow(r) {
for (const db of r.databases) checks.push(db.engine === 'postgresql'
? `\`pg_restore --list\` (${db.name}) = ${db.integrity === 'pg_restore --list ok' ? 'ok' : cell(db.integrity).slice(0, 80)}`
: `\`pragma integrity_check\` (${db.name}) = ${db.integrity === 'ok' ? 'ok' : cell(db.integrity).slice(0, 80)}`);
++ for (const o of r.objects || []) checks.push(o.integrity === 'sha256 ok'
++ ? `object store \`${o.name}\` ${o.blobs} blob(s) sha256 ok`
++ : `object store \`${o.name}\`: ${cell(o.integrity).slice(0, 80)}`);
if (r.ready) checks.push(r.ready.ok ? `drill instance \`${new URL(r.ready.url).pathname}\` ${r.ready.status} after ${r.ready.seconds}s` : `drill instance not ready (${r.ready.status || 'no answer'})`);
const same = r.compare.filter((c) => c.match).map((c) => `\`${c.path}\``);
const diff = r.compare.filter((c) => !c.match).map((c) => `\`${c.path}\` differs (${cell(c.detail)})`);
@@@ -603,4 +729,4 @@ function markdownRow(r) {
return `| ${when} | ${r.service} | \`${r.backup}/\` (\`ovhost drill ${r.service}\`) | ${checks.join('; ')} | ${r.result} |`;
}
--module.exports = { drill, DrillError, compareBodies };
++module.exports = { drill, DrillError, compareBodies, verifyObjectBlobs, tarMemberProblem };
ddiff --git a/lib/inventory.js b/lib/inventory.js
iindex 996b009..2c9a30e 100644
---- a/lib/inventory.js
++++ b/lib/inventory.js
@@@ -143,6 +143,26 @@ function normalisePreflight(raw, where) {
return out;
}
++/**
++ * `objects`: directories of content-addressed blobs the service owns OUTSIDE its databases, such as
++ * OpenVibe.Host's tenant object store (host.example.json: `HOST_STORAGE_DIR`). Each regular file in
++ * the directory is named by its own sha256, so `ovhost backup` can archive the tree as one artifact
++ * and a drill can check every blob's sha256 after restoring it.
++ */
++function normaliseObjects(raw, where) {
++ if (raw == null) return [];
++ if (!Array.isArray(raw)) throw new InventoryError(`${where}.objects must be a list of directories`);
++ const seen = new Set();
++ return raw.map((o, i) => {
++ const at = `${where}.objects[${i}]`;
++ if (!o || typeof o !== 'object' || Array.isArray(o)) throw new InventoryError(`${at} must be an object`);
++ if (!FILE_NAME_RE.test(o.name || '')) throw new InventoryError(`${at}.name must be a plain name`);
++ if (seen.has(o.name)) throw new InventoryError(`${at}.name "${o.name}" is listed twice`);
++ seen.add(o.name);
++ return { name: o.name, path: abs(o.path, `${at}.path`) };
++ });
++}
++
function normaliseService(id, raw, defaults) {
if (!ID_RE.test(id)) throw new InventoryError(`service id "${id}" must match ${ID_RE}`);
const where = `services.${id}`;
@@@ -236,6 +256,9 @@ function normaliseService(id, raw, defaults) {
}
throw new InventoryError(`${at}.engine must be sqlite or postgresql`);
}),
++ // Content-addressed object directories alongside the databases (tenant objects, Host's
++ // HOST_STORAGE_DIR). Copied as one archive per directory; never opened as a database.
++ objects: normaliseObjects(s.objects, where),
backupOnChange: s.backupOnChange || [],
nginx: s.nginx || null,
// The public origin release notifications name (lib/announce.js); default: the manifest's publicOrigin.
ddiff --git a/lib/offsite.js b/lib/offsite.js
iindex 1eb8b68..cb3f833 100644
---- a/lib/offsite.js
++++ b/lib/offsite.js
@@@ -1,10 +1,10 @@
'use strict';
/**
-- * Off-host copies of `ovhost backup --all` runs: every backed-up database is encrypted on the host
-- * (lib/backup-crypto.js, AES-256-GCM with a key that stays on the host) and uploaded to S3-compatible
-- * object storage (Backblaze B2 in production, the same API Media uses). Nothing is uploaded in the
-- * clear except the run manifest, which holds names, sizes and hashes, never data or secrets, and
-- * carries an HMAC made with the backup key.
++ * Off-host copies of `ovhost backup --all` runs: every backed-up file (a database copy or an object
++ * archive) is encrypted on the host (lib/backup-crypto.js, AES-256-GCM with a key that stays on the
++ * host) and uploaded to S3-compatible object storage (Backblaze B2 in production, the same API Media
++ * uses). Nothing is uploaded in the clear except the run manifest, which holds names, sizes and hashes,
++ * never data or secrets, and carries an HMAC made with the backup key.
*
* Configuration comes from a root-only env file (default /etc/openvibe/backup.env):
*
@@@ -252,7 +252,7 @@ async function push(ctx, summary, { cfg, client, partSize = PART_SIZE, chunkSize
await pipeline(exec.readStream(f.dest), plain, bcrypto.createEncryptStream(master, chunkSize ? { chunkSize } : {}), cipher, async (source) => { up = await uploadIterable(client, cfg.bucket, key, source, { partSize }); });
const p = plain.result();
const c = cipher.result();
-- const entry = { service: svc.service, name: f.name, file: name, object: key, source: f.source || null, plainBytes: p.bytes, plainSha256: p.sha256, cipherBytes: c.bytes, cipherSha256: c.sha256, parts: up.parts, seconds: Math.round((exec.now() - t0) / 1000) };
++ const entry = { service: svc.service, name: f.name, kind: f.kind || 'database', file: name, object: key, source: f.source || null, plainBytes: p.bytes, plainSha256: p.sha256, cipherBytes: c.bytes, cipherSha256: c.sha256, parts: up.parts, seconds: Math.round((exec.now() - t0) / 1000) };
result.uploaded.push(entry);
(manifest.services[svc.service] = manifest.services[svc.service] || { files: [] }).files.push(entry);
log(`uploaded ${f.dest} → s3://${cfg.bucket}/${key} (${c.bytes} bytes encrypted)`);
@@@ -322,7 +322,7 @@ async function readManifest(client, cfg, master, host, run) {
return m;
}
--/** Directories restore-download must never write into: anything holding a declared database. */
++/** Directories restore-download must never write into: anything holding a declared database or object directory. */
function unsafeOut(inv, out) {
for (const svc of Object.values(inv.services)) {
for (const db of svc.databases) {
@@@ -331,6 +331,11 @@ function unsafeOut(inv, out) {
const dir = path.dirname(db.path);
if (out === dir || db.path.startsWith(`${out}/`) || out.startsWith(`${dir}/`)) return `${out} holds or sits inside ${svc.id}'s database directory ${dir}`;
}
++ // An object directory is itself a directory the live service writes: never restore over it.
++ for (const obj of svc.objects || []) {
++ const dir = obj.path;
++ if (out === dir || dir.startsWith(`${out}/`) || out.startsWith(`${dir}/`)) return `${out} holds or sits inside ${svc.id}'s object directory ${dir}`;
++ }
if (out === svc.repo || out.startsWith(`${svc.repo}/`) || svc.repo.startsWith(`${out}/`)) return `${out} overlaps ${svc.id}'s checkout ${svc.repo}`;
}
for (const d of [inv.backupDir, inv.stateDir]) if (out === d || d.startsWith(`${out}/`) || out.startsWith(`${d}/`)) return `${out} overlaps ${d}`;
@@@ -390,18 +395,25 @@ async function restoreDownload(ctx, id, runArg, { cfg, client, out = null, host:
}
// A PostgreSQL artifact is a pg_dump archive, checked by parsing it (pg_restore --list, which
// needs no database and is run as the caller — the copy is root-only 0600, and postgres could
-- // not read it); a SQLite copy is quick_checked through the worker.
++ // not read it); a SQLite copy is quick_checked through the worker. An object-directory archive
++ // is neither: its own sha256 has already matched the manifest, and a drill is what extracts it
++ // and checks every blob.
let check;
const isDump = path.extname(f.file) === '.dump';
-- try {
-- if (isDump) { await dbengine.verifyDump(exec, dest, { as: null }); check = 'pg_restore --list ok'; }
-- else {
-- const rows = await exec.sqlite(dest, 'PRAGMA quick_check');
-- check = rows && rows[0] ? String(Object.values(rows[0])[0]) : 'no answer';
-- }
-- } catch (err) { check = `error: ${err.message}`; }
-- record.files.push({ name: f.name, file: dest, source: f.source, bytes: p.bytes, sha256: p.sha256, quickCheck: check });
-- if (isDump ? check.startsWith('error') : check !== 'ok') throw new OffsiteError(`${dest}: ${check}`);
++ const isObjects = f.kind === 'objects' || f.file.endsWith('.tar.gz');
++ if (isObjects) {
++ check = 'object archive (verify with a drill)';
++ } else {
++ try {
++ if (isDump) { await dbengine.verifyDump(exec, dest, { as: null }); check = 'pg_restore --list ok'; }
++ else {
++ const rows = await exec.sqlite(dest, 'PRAGMA quick_check');
++ check = rows && rows[0] ? String(Object.values(rows[0])[0]) : 'no answer';
++ }
++ } catch (err) { check = `error: ${err.message}`; }
++ }
++ record.files.push({ name: f.name, kind: isObjects ? 'objects' : 'database', file: dest, source: f.source, bytes: p.bytes, sha256: p.sha256, quickCheck: isObjects ? null : check });
++ if (!isObjects && (isDump ? check.startsWith('error') : check !== 'ok')) throw new OffsiteError(`${dest}: ${check}`);
}
return record;
}
ddiff --git a/test/backup-offsite.test.js b/test/backup-offsite.test.js
iindex c1672bc..f4326ca 100644
---- a/test/backup-offsite.test.js
++++ b/test/backup-offsite.test.js
@@@ -15,10 +15,12 @@ const retention = require('../lib/retention');
const offsite = require('../lib/offsite');
const { normalise } = require('../lib/inventory');
const { mockS3 } = require('./s3-mock');
++const { readTarGz } = require('../lib/tar');
const DAY = 86400000;
const KEY_HEX = crypto.createHash('sha256').update('test backup key').digest('hex');
const S3_SECRET = `${SECRET}-s3`;
++const TENANT_BLOB = 'tenant object bytes\n';
/** The standard fake host plus AI (two databases) and OpenRe (not deployed), a backup env and key. */
function backupHost({ envMode = 0o600, keyMode = 0o600, envOwner = 'root', env = null } = {}) {
@@@ -75,6 +77,25 @@ function noSecrets(text, what) {
for (const s of [S3_SECRET, SECRET, KEY_HEX]) assert.ok(!String(text).includes(s), `${what} leaked a secret value`);
}
++/** backupHost plus a service "host" on PostgreSQL with a declared content-addressed object directory. */
++function objectsHost() {
++ const host = backupHost();
++ const doc = JSON.parse(host.read('/etc/openvibe/host.json'));
++ doc.services.host = {
++ repo: '/opt/openvibe.host',
++ units: [],
++ databases: [{ name: 'host', engine: 'postgresql', database: 'ov_host' }],
++ objects: [{ name: 'objects', path: '/var/lib/openvibe-host-api/objects' }],
++ };
++ host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });
++ host.inv = normalise(doc);
++ host.pgDatabases.add('ov_host');
++ host.pgBackups = [{ type: 'diff', timestamp: { stop: Math.floor(host.exec.now() / 1000) - 3600 } }];
++ host.tenantSha = crypto.createHash('sha256').update(TENANT_BLOB).digest('hex');
++ host.put(`/var/lib/openvibe-host-api/objects/projects/prj_1/${host.tenantSha.slice(0, 2)}/${host.tenantSha}`, TENANT_BLOB, { owner: 'ubuntu', mode: 0o644 });
++ return host;
++}
++
async function decrypt(key, buf) {
const chunks = [];
await pipeline(Readable.from([buf]), bcrypto.createDecryptStream(key), async (src) => { for await (const c of src) chunks.push(c); });
@@@ -355,6 +376,83 @@ runTests([
assert.strictEqual(s3.objects.size, 0);
}),
++ test('--offsite: a declared object directory is archived, encrypted, uploaded and recorded in the manifest', async () => {
++ const host = objectsHost();
++ const r = await host.cli('backup', '--all', '--offsite');
++ assert.strictEqual(r.code, 0, r.out);
++ // The archive is an ordinary root-only copy in the local backup...
++ const local = host.files.get('/var/backups/openvibe/host/20260922-120000/objects.tar.gz');
++ assert.deepStrictEqual([local.owner, local.mode], ['root', 0o600]);
++ // ...and an encrypted object off-host, under the same rules as every other file.
++ const base = 'openvibe-backups/fake-host/20260922-120000/';
++ const key = `${base}host/objects.tar.gz.ovbk`;
++ assert.ok(host.s3.objects.has(key), [...host.s3.objects.keys()].join('\n'));
++ const enc = host.s3.objects.get(key);
++ assert.ok(!enc.includes('tenant object'), 'the archive is not encrypted');
++ const plain = await decrypt(Buffer.from(KEY_HEX, 'hex'), enc);
++ const entries = readTarGz(plain);
++ assert.deepStrictEqual(entries.map((e) => e.name), [`projects/prj_1/${host.tenantSha.slice(0, 2)}/${host.tenantSha}`]);
++ assert.strictEqual(entries[0].content, TENANT_BLOB);
++ const m = JSON.parse(host.s3.objects.get(`${base}manifest.json`));
++ const f = m.services.host.files.find((x) => x.file === 'objects.tar.gz');
++ assert.deepStrictEqual([f.kind, f.name], ['objects', 'objects']);
++ assert.strictEqual(f.plainSha256, crypto.createHash('sha256').update(plain).digest('hex'));
++ noSecrets(host.s3.objects.get(`${base}manifest.json`), 'the manifest');
++ noSecrets(JSON.stringify(host.s3.sent), 'S3 requests');
++ noSecrets(r.out, 'CLI output');
++ }),
++
++ test('an object archive is written only after root has taken the staging directory over', async () => {
++ const host = objectsHost();
++ const r = await host.cli('backup', 'host');
++ assert.strictEqual(r.code, 0, r.out);
++ const tarCall = host.calls.find((c) => c.cmd === 'tar' && c.args.some((a) => String(a).endsWith('objects.tar.gz')));
++ assert.ok(tarCall, 'the object directory was tarred');
++ assert.strictEqual(tarCall.privileged, true);
++ const stage = path.dirname(tarCall.args[1]);
++ const lockAt = host.calls.findIndex((c) => c.cmd === 'chown' && c.args[c.args.length - 1] === stage && c.args.includes('root:root'));
++ const chmodAt = host.calls.findIndex((c) => c.cmd === 'chmod' && c.args[c.args.length - 1] === stage && c.args.includes('0700'));
++ const tarAt = host.calls.indexOf(tarCall);
++ assert.ok(lockAt >= 0 && chmodAt >= 0, 'root locked the staging directory down');
++ assert.ok(lockAt < tarAt && chmodAt < tarAt, 'root owned the stage before tar opened the archive');
++ }),
++
++ test('a tar warning for transient scratch under the store is accepted, and the scratch excluded', async () => {
++ const host = objectsHost();
++ host.put('/var/lib/openvibe-host-api/objects/tmp/aaaaaaaaaaaa.part', 'half-written upload', { owner: 'ubuntu', mode: 0o644 });
++ host.onTar = (args) => (String(args[0]).includes('c') ? { code: 1, stderr: 'tar: ./tmp/aaaaaaaaaaaa.part: file changed as we read it\n' } : undefined);
++ const r = await host.cli('backup', 'host', '--json');
++ host.onTar = null;
++ assert.strictEqual(r.code, 0, r.out);
++ const rec = JSON.parse(r.out);
++ const f = rec.files.find((x) => x.kind === 'objects');
++ assert.ok(f && !f.error, JSON.stringify(f));
++ assert.ok(Number.isFinite(f.sourceBytes) && f.sourceBytes > 0, 'the source size was recorded for the drill');
++ const tarCall = host.calls.find((c) => c.cmd === 'tar' && c.args.some((a) => String(a).endsWith('.tar.gz')));
++ assert.ok(tarCall.args.includes('--exclude=./tmp'), tarCall.args.join(' '));
++ const names = readTarGz(host.files.get(`${rec.dir}/objects.tar.gz`).content).map((e) => e.name);
++ assert.ok(!names.some((n) => n.startsWith('tmp/')), `scratch was archived: ${names.join(', ')}`);
++ assert.ok(names.length >= 1, 'the tenant blob is still archived');
++ }),
++
++ test('a genuine tar failure still fails the backup', async () => {
++ const host = objectsHost();
++ host.onTar = (args) => (String(args[0]).includes('c') ? { code: 2, stderr: 'tar: ./projects: Cannot open: Permission denied\n' } : undefined);
++ const r = await host.cli('backup', 'host');
++ host.onTar = null;
++ assert.strictEqual(r.code, 2, r.out);
++ assert.match(r.out, /Permission denied/);
++ }),
++
++ test('an object directory is not archived when the staging filesystem has no room', async () => {
++ const host = objectsHost();
++ host.statfsFree = 1024;
++ const r = await host.cli('backup', 'host');
++ assert.strictEqual(r.code, 2, r.out);
++ assert.match(r.out, /not enough free space under \/var\/backups\/openvibe\.staging/);
++ assert.ok(!host.calls.some((c) => c.cmd === 'tar'), 'nothing was tarred');
++ }),
++
test('off-host retention: runs older than 30 days are deleted, the newest 7 always stay, and nothing is pruned after a failed upload', async () => {
const host = backupHost();
const put = (run) => {
@@@ -487,6 +585,22 @@ runTests([
assert.match(refused.out, /pg_restore --list .* failed|text format dump/);
}),
++ test('restore-download: an object archive is written without a database check and never over the live object directory', async () => {
++ const host = objectsHost();
++ await host.cli('backup', '--all', '--offsite');
++ const near = await host.cli('restore-download', 'host', 'latest', '--out', '/var/lib/openvibe-host-api/objects');
++ assert.strictEqual(near.code, 1, near.out);
++ assert.match(near.out, /refusing --out: .*object directory/);
++ const r = await host.cli('restore-download', 'host', 'latest', '--out', '/root/r-host', '--json');
++ assert.strictEqual(r.code, 0, r.out);
++ const rec = JSON.parse(r.out);
++ const f = rec.files.find((x) => x.file.endsWith('objects.tar.gz'));
++ assert.deepStrictEqual([f.kind, f.quickCheck], ['objects', null], 'an object archive is not quick_checked as a database');
++ assert.ok(host.files.has('/root/r-host/objects.tar.gz'));
++ assert.ok(!host.sqliteCalls.some((c) => c.db === '/root/r-host/objects.tar.gz'));
++ assert.ok(rec.files.some((x) => x.file.endsWith('host.dump') && x.kind === 'database'), 'the database is still checked');
++ }),
++
// ── configuration ──────────────────────────────────────────────────────────
test('config: root-only env file and key; missing names listed by name; not root refused; no value ever printed', async () => {
let host = backupHost({ envMode: 0o644 });
ddiff --git a/test/drill.test.js b/test/drill.test.js
iindex f44a777..c972f42 100644
---- a/test/drill.test.js
++++ b/test/drill.test.js
@@@ -6,16 +6,24 @@
* the drill log is written.
*/
const assert = require('assert');
++const crypto = require('crypto');
const fs = require('fs');
const path = require('path');
const { scenario, test, runTests, SECRET } = require('./helpers');
const { normalise } = require('../lib/inventory');
--const { compareBodies } = require('../lib/commands/drill');
++const { compareBodies, verifyObjectBlobs } = require('../lib/commands/drill');
const DRILL_DIR = '/var/lib/openvibe-drills';
const PROD_DB = '/var/lib/openvibe-tools/tools.db';
++const OBJECTS_DIR = '/var/lib/openvibe-host-api/objects';
const PASTES = '[{"id":"pst_1","title":"hello"},{"id":"pst_2","title":"world"}]';
++/** A content-addressed blob under the Host object-store layout: <root>/projects/<prj>/<aa>/<sha256>. */
++function blob(root, data) {
++ const sha = crypto.createHash('sha256').update(data).digest('hex');
++ return { sha, path: `${root}/projects/prj_1/${sha.slice(0, 2)}/${sha}` };
++}
++
function toolsEntry(drillOverrides = {}) {
return {
repo: '/opt/openvibe.tools',
@@@ -40,10 +48,12 @@ function toolsEntry(drillOverrides = {}) {
* The standard fake host plus Tools in production, with one `ovhost backup tools` taken.
* The drill instance's behaviour is controlled through `host.drill`.
*/
--async function drillScenario({ drill: drillOverrides, backup = true } = {}) {
++async function drillScenario({ drill: drillOverrides, backup = true, objects = null, blobs = [] } = {}) {
const host = scenario();
const doc = JSON.parse(host.read('/etc/openvibe/host.json'));
-- doc.services.tools = toolsEntry(drillOverrides);
++ const tools = toolsEntry(drillOverrides);
++ if (objects) tools.objects = objects;
++ doc.services.tools = tools;
host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });
const repo = host.createRepo('/opt/openvibe.tools', { owner: 'ubuntu' });
@@@ -100,6 +110,8 @@ async function drillScenario({ drill: drillOverrides, backup = true } = {}) {
return undefined;
};
++ for (const b of blobs) host.put(b.path, b.content, { owner: 'ubuntu', mode: 0o644 });
++
if (backup) {
const b = await host.cli('backup', 'tools', '--json');
assert.strictEqual(b.code, 0, b.out);
@@@ -545,6 +557,137 @@ runTests([
normalise({ services: { apps: multi } }); // valid inventory; the drill itself refuses (below)
}),
++ test('objects: the declared object directory is archived, restored into scratch, and every blob sha256 checked', async () => {
++ const a = 'tenant page a\n';
++ const b = 'tenant asset b\n';
++ const pa = blob(OBJECTS_DIR, a);
++ const pb = blob(OBJECTS_DIR, b);
++ const host = await drillScenario({
++ objects: [{ name: 'objects', path: OBJECTS_DIR }],
++ blobs: [{ path: pa.path, content: a }, { path: pb.path, content: b }],
++ drill: {
++ env: { PORT: '{port}', HOST: '127.0.0.1', DISCORD_RELAY_ENABLED: '0', EVENTS_URL: '', UPLOAD_DIR: '{tmp}/data/uploads', GREETING: 'two words', HOST_STORAGE_DIR: '{tmp}/objects' },
++ dirs: ['{tmp}/data', '{tmp}/objects'],
++ },
++ });
++ const archive = `${host.backupDir}/objects.tar.gz`;
++ assert.deepStrictEqual([host.files.get(archive).owner, host.files.get(archive).mode], ['root', 0o600], 'the archive is an ordinary root-only copy');
++ const r = await host.cli('drill', 'tools', '--keep');
++ assert.strictEqual(r.code, 0, r.out);
++ const rec = lastLog(host);
++ assert.strictEqual(rec.kept, true);
++ assert.deepStrictEqual(rec.objects.map((o) => [o.name, o.blobs, o.skipped, o.integrity]), [['objects', 2, 0, 'sha256 ok']]);
++ const tmp = rec.dir;
++ // The archive is listed first, then extracted without restoring production's ownership or modes.
++ assert.ok(host.calls.some((c) => c.cmd === 'tar' && c.args[0] === '-tvzf' && c.args[1] === archive), 'listed before extraction');
++ const extract = host.calls.find((c) => c.cmd === 'tar' && c.args[0] === '-xzf');
++ assert.deepStrictEqual(extract.args, ['-xzf', archive, '-C', `${tmp}/objects`, '--no-same-owner', '--no-same-permissions']);
++ assert.strictEqual(extract.privileged, true);
++ // Restored into the drill's own scratch directory, never the live object directory.
++ assert.strictEqual(rec.objects[0].copy, `${tmp}/objects`);
++ assert.ok(rec.objects[0].copy.startsWith(`${tmp}/`));
++ assert.strictEqual(host.read(`${tmp}/objects/projects/prj_1/${pa.sha.slice(0, 2)}/${pa.sha}`), a);
++ assert.strictEqual(host.read(`${tmp}/objects/projects/prj_1/${pb.sha.slice(0, 2)}/${pb.sha}`), b);
++ // The instance is pointed at the restored directory, not production's.
++ assert.ok(host.read(`${tmp}/drill.env`).split('\n').includes(`HOST_STORAGE_DIR=${tmp}/objects`));
++ assert.match(rec.markdown, /object store `objects` 2 blob\(s\) sha256 ok/);
++ // The live objects were never written or moved.
++ assert.strictEqual(host.read(pa.path), a);
++ assert.strictEqual(host.read(pb.path), b);
++ }),
++
++ test('objects: a blob whose bytes do not match the sha256 in its name fails the drill before anything starts', async () => {
++ const good = 'good blob\n';
++ const p = blob(OBJECTS_DIR, good);
++ const wrongName = 'a'.repeat(64);
++ const host = await drillScenario({
++ objects: [{ name: 'objects', path: OBJECTS_DIR }],
++ blobs: [{ path: p.path, content: good }, { path: `${OBJECTS_DIR}/projects/prj_1/aa/${wrongName}`, content: 'not its name\n' }],
++ });
++ const before = fsSnapshot(host);
++ const r = await host.cli('drill', 'tools');
++ assert.strictEqual(r.code, 2, r.out);
++ const rec = lastLog(host);
++ assert.strictEqual(rec.failure.stage, 'objects');
++ assert.match(rec.failure.message, /blob\(s\) do not match their name/);
++ assert.match(rec.failure.message, /aa\/a{64}/);
++ assert.strictEqual(host.systemdRuns.length, 0, 'nothing is started');
++ assert.ok(!host.files.has(rec.dir));
++ assert.strictEqual(host.read(p.path), good, 'the live object is untouched');
++ for (const changed of changedPaths(before, fsSnapshot(host))) assert.ok(allowedWrite(changed, rec.dir), `unexpected write: ${changed}`);
++ }),
++
++ test('objects: a service declaring an object directory refuses to drill from a backup without its archive', async () => {
++ // The directory does not exist, so the backup records the object directory as skipped.
++ const host = await drillScenario({ objects: [{ name: 'objects', path: OBJECTS_DIR }], blobs: [] });
++ const r = await host.cli('drill', 'tools');
++ assert.strictEqual(r.code, 1);
++ assert.match(r.out, /has no copy of the objects object directory \(objects\.tar\.gz\)/);
++ assert.strictEqual(host.systemdRuns.length, 0);
++ }),
++
++ test('objects: a symlink or special-file member is refused before root extracts the archive', async () => {
++ for (const member of [{ type: 'l', name: './evil', target: '/' }, { type: 'p', name: './fifo' }]) {
++ const a = 'tenant page a\n';
++ const pa = blob(OBJECTS_DIR, a);
++ const host = await drillScenario({ objects: [{ name: 'objects', path: OBJECTS_DIR }], blobs: [{ path: pa.path, content: a }] });
++ host.tarMembers = [member];
++ const r = await host.cli('drill', 'tools');
++ assert.strictEqual(r.code, 2, `${member.type}: ${r.out}`);
++ const rec = lastLog(host);
++ assert.strictEqual(rec.failure.stage, 'objects');
++ assert.match(rec.failure.message, /not a regular file or directory/);
++ assert.ok(!host.calls.some((c) => c.cmd === 'tar' && c.args[0] === '-xzf'), `${member.type}: never extracted`);
++ assert.ok(!host.files.has(rec.dir), 'cleaned up');
++ }
++ }),
++
++ test('objects: a symlinked directory already in the restored tree is refused, not walked as root', async () => {
++ const host = scenario();
++ const dir = '/var/lib/openvibe-drills/tools-20260922-120100/objects';
++ const good = 'a good blob\n';
++ const sha = crypto.createHash('sha256').update(good).digest('hex');
++ host.put(`${dir}/projects/prj_1/${sha.slice(0, 2)}/${sha}`, good, { owner: 'ubuntu' });
++ // A file whose name IS a sha256 but whose bytes are not, behind a symlinked directory: if the
++ // walk followed the link it would hash this and report a mismatch.
++ const outsideSha = 'b'.repeat(64);
++ host.put(`/var/lib/openvibe-secret/${outsideSha}`, 'not its name\n', { owner: 'root', mode: 0o600 });
++ await host.exec.symlink('/var/lib/openvibe-secret', `${dir}/evil`);
++ const entry = { blobs: 0, skipped: 0, bytes: 0, mismatches: 0 };
++ await assert.rejects(
++ verifyObjectBlobs(host.exec, dir, entry),
++ (err) => /evil \(symlink or special file\)/.test(err.message) && !err.message.includes(outsideSha) && !err.message.includes('/var/lib/openvibe-secret'),
++ );
++ assert.strictEqual(entry.blobs, 1, 'the real blob was still checked');
++ assert.strictEqual(entry.mismatches, 1);
++ }),
++
++ test('objects: a drill with no room to extract the archive fails before root restores it', async () => {
++ const a = 'tenant page a\n';
++ const pa = blob(OBJECTS_DIR, a);
++ const host = await drillScenario({ objects: [{ name: 'objects', path: OBJECTS_DIR }], blobs: [{ path: pa.path, content: a }] });
++ const callsBefore = host.calls.length;
++ host.statfsFree = 1024;
++ const r = await host.cli('drill', 'tools');
++ assert.strictEqual(r.code, 2, r.out);
++ const rec = lastLog(host);
++ assert.strictEqual(rec.failure.stage, 'space');
++ assert.match(rec.failure.message, /not enough free space under \/var\/lib\/openvibe-drills/);
++ assert.ok(!host.calls.slice(callsBefore).some((c) => c.cmd === 'tar'), 'nothing was tarred or extracted');
++ assert.ok(!host.files.has(rec.dir));
++ }),
++
++ test('inventory: object directories are validated', () => {
++ const base = { owner: 'ubuntu', repo: '/opt/openvibe.host', units: [], databases: [{ name: 'host', engine: 'postgresql', database: 'ov_host' }] };
++ const svc = (objects) => normalise({ services: { h: { ...base, objects } } });
++ assert.deepStrictEqual(svc([{ name: 'objects', path: '/var/lib/openvibe-host-api/objects' }]).services.h.objects, [{ name: 'objects', path: '/var/lib/openvibe-host-api/objects' }]);
++ assert.deepStrictEqual(svc(undefined).services.h.objects, []);
++ assert.throws(() => svc([{ name: 'a/b', path: '/x' }]), /name must be a plain name/);
++ assert.throws(() => svc([{ name: 'objects', path: 'relative' }]), /absolute path/);
++ assert.throws(() => svc([{ name: 'objects', path: '/x' }, { name: 'objects', path: '/y' }]), /listed twice/);
++ assert.throws(() => svc('nope'), /must be a list of directories/);
++ }),
++
test('several units and neither drill.unit nor drill.command: refused', async () => {
const host = await appsScenario({ drill: { unit: null } });
const r = await host.cli('drill', 'apps');
ddiff --git a/test/fake-host.js b/test/fake-host.js
iindex ab25574..843d063 100644
---- a/test/fake-host.js
++++ b/test/fake-host.js
@@@ -13,6 +13,7 @@
const crypto = require('crypto');
const path = require('path');
const { Readable, Writable } = require('stream');
++const { createTarGz, readTarGz } = require('../lib/tar');
function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', start = Date.parse('2026-09-22T12:00:00Z') } = {}) {
const files = new Map(); // path -> { type: 'file'|'dir'|'symlink', content, mode, owner, target }
@@@ -53,6 +54,8 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', st
statfsFree: 100 * 1024 ** 3,
statfsSize: 500 * 1024 ** 3,
reads: [],
++ onTar: null, // (args, opts, real) -> undefined | { code, stdout, stderr } ; override a tar result
++ tarMembers: [], // [{ type, name, target }] extra (non-)members a `tar -tvzf` listing reports
certbot: null, // (args, opts) -> { code, stdout, stderr } | Error ; default: nothing to do
certbotCalls: [], // { args, as, privileged }
journalctl: null, // (args, opts) -> { code, stdout, stderr } ; default: no entries
@@@ -365,6 +368,66 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', st
return { code: 0, stdout: 'Done', stderr: '' };
}
++ // ── tar (backup/drill of a content-addressed object directory) ──
++ // The forms the backup and drill use: `tar -czf <dest> -C <dir> .`, `tar -tvzf <archive>` (the
++ // drill's member pre-check) and `tar -xzf <archive> -C <dir>`. The archive is a real gzipped tar
++ // (lib/tar.js), so it survives the off-host encrypt/decrypt pipeline byte for byte.
++ function tarCmd(args, opts) {
++ const result = tarCmdReal(args, opts);
++ // A test can override the result (to simulate a warning or a failure) after the real archive
++ // was made, so --exclude and the archive's contents stay realistic.
++ if (host.onTar) { const r = host.onTar(args, opts, result); if (r) return { stdout: '', stderr: '', ...r }; }
++ return result;
++ }
++ function tarCmdReal(args, opts) {
++ const mode = String(args[0] || '');
++ const archive = args[1];
++ const ci = args.indexOf('-C');
++ const dir = ci >= 0 ? args[ci + 1] : '.';
++ // `--exclude=PAT` (used for Host's transient <root>/tmp scratch): skip a member whose relative
++ // name is the pattern or sits under it.
++ const excludes = args.filter((a) => a.startsWith('--exclude=')).map((a) => a.slice(10).replace(/^\.\//, ''));
++ const excluded = (name) => excludes.some((ex) => name === ex || name.startsWith(`${ex}/`));
++ if (mode.includes('c')) {
++ const target = args[args.length - 1];
++ const base = resolveLinks(dir);
++ const root = resolveLinks(path.join(dir, target));
++ const entries = [];
++ for (const [k, v] of files) {
++ if (v.type !== 'file') continue;
++ if (k !== root && !k.startsWith(`${root}/`)) continue;
++ const name = path.relative(base, k);
++ if (excluded(name)) continue;
++ entries.push({ name, content: v.content });
++ }
++ put(archive, createTarGz(entries, clock), { owner: opts.as || user, mode: 0o600 });
++ return { code: 0, stdout: '', stderr: '' };
++ }
++ if (mode.includes('t')) {
++ const e = get(archive);
++ if (!e || e.type !== 'file') return { code: 1, stdout: '', stderr: `tar: ${archive}: No such file or directory` };
++ let parsed;
++ try { parsed = readTarGz(Buffer.isBuffer(e.content) ? e.content : Buffer.from(String(e.content))); } catch (err) { return { code: 1, stdout: '', stderr: `tar: ${err.message}` }; }
++ const lines = parsed.map((f) => `-rw-r--r-- ubuntu/ubuntu ${Buffer.byteLength(f.content)} 2026-09-22 12:00 ${f.name}`);
++ for (const m of host.tarMembers) lines.push(`${m.type}rw-r--r-- root/root 0 2026-09-22 12:00 ${m.name}${m.target ? ` -> ${m.target}` : ''}`);
++ return { code: 0, stdout: lines.length ? `${lines.join('\n')}\n` : '', stderr: '' };
++ }
++ if (mode.includes('x')) {
++ const e = get(archive);
++ if (!e || e.type !== 'file') return { code: 1, stdout: '', stderr: `tar: ${archive}: No such file or directory` };
++ const destdir = resolveLinks(dir);
++ let parsed;
++ try { parsed = readTarGz(Buffer.isBuffer(e.content) ? e.content : Buffer.from(String(e.content))); } catch (err) { return { code: 1, stdout: '', stderr: `tar: ${err.message}` }; }
++ for (const f of parsed) {
++ const name = f.name.replace(/^\.\//, '');
++ if (!name || name.split('/').includes('..')) continue;
++ put(path.join(destdir, name), f.content, { owner: opts.as || 'root', mode: 0o644 });
++ }
++ return { code: 0, stdout: '', stderr: '' };
++ }
++ return { code: 1, stdout: '', stderr: `fake tar: unsupported ${args.join(' ')}` };
++ }
++
// ── systemd ──
host.addUnit = (unit, state = {}) => {
units.set(unit, { load: 'loaded', active: 'active', sub: 'running', mainPid: 1000 + units.size, fragmentPath: `/etc/systemd/system/${unit}`, dropIns: [], partOf: [], restarts: 0, runningSha: null, ...state });
@@@ -499,6 +562,17 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', st
const r = host.onNode ? host.onNode(args, opts) : undefined;
return r ? { stdout: '', stderr: '', ...r } : { code: 0, stdout: '', stderr: '' };
}
++ case 'du': {
++ const target = args[args.length - 1];
++ const root = resolveLinks(target);
++ if (!files.has(root)) return { code: 1, stdout: '', stderr: `du: cannot access '${target}': No such file or directory` };
++ let total = 0;
++ for (const [k, v] of files) {
++ if (k !== root && !k.startsWith(`${root}/`)) continue;
++ total += v.type === 'file' ? Buffer.byteLength(Buffer.isBuffer(v.content) ? v.content : String(v.content)) : 4096;
++ }
++ return { code: 0, stdout: `${total}\t${target}\n`, stderr: '' };
++ }
case 'certbot': {
host.certbotCalls.push({ args, as: opts.as || null, privileged: !!opts.privileged });
const r = host.certbot ? host.certbot(args, opts) : undefined;
@@@ -572,6 +646,7 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', st
return { code: 0, stdout: `${JSON.stringify(stanzas)}\n`, stderr: '' };
}
case 'systemd-run': return systemdRunCmd(args);
++ case 'tar': return tarCmd(args, opts);
case 'cp': {
const src = args[args.length - 2];
const dest = args[args.length - 1];