openvibers/openvibe.host Commits

Scheduled, root-only and off-host backups: ovhost backup --all (every service with databases, failure-isolated, 7 daily + 4 weekly retention, JSON run summary), encrypted off-host copies to S3-compatible storage (Node crypto AES-256-GCM, @aws-sdk/client-s3, 30-day remote retention), restore-download, and a daily openvibe-backup.timer

OP OpenVibers committed 23 Sept 2026 · 34d706c7ee · 25 files, +2504 −44

- backup --all: one failing service never stops the others; exit 2 if anything failed. Every
  database is checked with quick_check. Summary in <stateDir>/backup-runs/<run>.json. A partly
  failed backup is recorded ok:false and never picked by a drill.
- Retention: newest good backup per day (7) and per ISO week (4). Only recorded stamp
  directories are pruned; failing days never age out the last good backups.
- Permissions: backups are root:root, directories 0700, files 0600. The worker (still the service
  user) writes into /var/backups/openvibe.staging/<svc>-<stamp>, switches the copy to
  journal_mode=DELETE so no -wal/-shm remain, and root takes the directory over, refuses links,
  chowns and renames it into place. Drills copy root-only backups with install -o <user> -m 0600.
- Off-host (--offsite, offsite push|list|check): streamed encrypt + upload (PutObject, or
  multipart with abort), HMAC-signed manifest, config from root-only /etc/openvibe/backup.env
  (BACKUP_S3_*, BACKUP_ENCRYPTION_KEY_FILE), never printed. Remote pruning runs only after a
  complete upload and always keeps the newest 7 runs.
- restore-download <service> <run|latest>: verifies the manifest HMAC and both hashes, decrypts
  into a new root-only directory, quick_checks; never overwrites and refuses any directory near
  a database or checkout.
- deploy/systemd/openvibe-backup.{service,timer}: daily 03:30 UTC, RandomizedDelaySec=15min.
- docs/backups.md: install steps, format, key escrow, bucket lifecycle, restore and drill of an
  off-host copy.

Changes

ddiff --git a/README.md b/README.md
iindex c3312de..51d72f8 100644
---- a/README.md
++++ b/README.md
@@@ -31,14 +31,14 @@ Stage A adds a few safety rules of its own:
## Owns
--- Stage A: the host inventory (services, units, env names, ports, probes, drain policy), environment validation, release install and rollback for git-checkout services, readiness and drain orchestration, the release log, certificate inventory, nginx vhost rendering and transactional install, config snapshots, SQLite backup hooks, restore drills.
++- Stage A: the host inventory (services, units, env names, ports, probes, drain policy), environment validation, release install and rollback for git-checkout services, readiness and drain orchestration, the release log, certificate inventory, nginx vhost rendering and transactional install, config snapshots, scheduled SQLite backups with retention and encrypted off-host copies, restore drills.
- Stage B: tenant projects (keyed by project id), static sites, immutable content-addressed deploy artifacts, activation and rollback, default and custom domains (DNS TXT verification), per-project quotas, upload/validation logs, the tenant vhosts (`ovhost nginx tenants`).
- Later (Stage C): sandbox profiles, budgets, secret references, outbound policy.
## Does not own
- Product business logic. Host calls product hooks (`build`) and reads product endpoints (`/api/ready`, `/api/streams`). It never embeds product rules.
--- Secrets. `ovhost` reads env files only to learn variable **names** and whether each one is empty. It never stores, prints or snapshots a value.
++- Secrets. `ovhost` reads service env files only to learn variable **names** and whether each one is empty. It never stores, prints or snapshots a value. The one exception is its own `/etc/openvibe/backup.env` (root-only): it reads the `BACKUP_*` values there to reach the backup bucket, and never prints or stores them.
- Platform-wide credentials for hosted code (never).
- Certificate issuance/renewal (certbot does this today). Stage A only takes inventory; Stage B never handles a certificate or key through its API.
- Projects as an identity concept: ADR-014 puts projects in OpenVibe.Network. Until Network has them, Host creates a `prj_` project for its owner and records `network_project_id` when one is given.
@@@ -76,6 +76,15 @@ What the tests demonstrate (`npm test`, every system call made against `test/fak
- `nginx -t` failures restore the previous vhost state. `certs` never reads a key file. Snapshots contain no secret values or remote-URL credentials.
- A **restore drill** restores the latest backup into a service-user-owned temp directory and requires `integrity_check = ok`. It starts a sandboxed second instance through `systemd-run` with the production env file plus an override file. It compares the declared paths and row counts with production, then stops the instance by its own pid and removes the directory. Failed integrity, a readiness timeout, an instance that dies, mismatches and a failed `systemd-run` are all reported, logged and cleaned up. It refuses to run without root or on a port already in use. A filesystem diff of the fake host shows it writes nothing outside the drill directory, its lock and its log, and it never reads the env file or passes on a secret-looking unit `Environment=`.
++- **Backups:**
++ - `backup --all` backs up every service with databases. One failing service leaves the others backed up and exits 2.
++ - Retention keeps the newest good backup of each of the last 7 days and 4 ISO weeks. A service whose backups keep failing never loses its last good ones. Directories ovhost did not record are never touched.
++ - Every backup directory is root 0700 and every copy is root 0600, with no `-wal`/`-shm`. A copy that is a link is refused.
++ - Off-host copies round-trip through AES-256-GCM. Tampering, truncation, swapped objects, a forged manifest and another key are all refused.
++ - Uploads go to a mocked S3 client (PutObject, or multipart with abort on failure). Off-host pruning never runs after a failed upload and always keeps the newest 7 runs.
++ - `restore-download` never overwrites and refuses any directory near a database or checkout.
++ - No secret value appears in any output, summary, request or manifest.
++
Not demonstrated yet: any of this on the production host. The Wave 21 exit criterion ("deploy, restart and roll back one service without interrupting unrelated runtimes or protected sessions") still needs a run on the host, with evidence.
## Using ovhost
@@@ -93,6 +102,9 @@ ovhost nginx render <service> [--variant http|sse|websocket] [--manifest <file>]
ovhost nginx tenants <service> [--wildcard-cert <name|dir>] [--install] Stage B tenant vhosts
ovhost snapshot <service> [--out <file>]
ovhost backup <service>
++ovhost backup --all [--offsite] [--no-prune] [--keep-daily <n>] [--keep-weekly <n>]
++ovhost offsite push [--run <run>] | offsite list [<service>] | offsite check
++ovhost restore-download <service> <run|latest> [--out <dir>] [--from-host <name>]
ovhost drill <service> [--backup <dir>] [--keep] restore drill (root only)
```
@@@ -111,6 +123,18 @@ Every command accepts `--json` and `--inventory <file>`. `drill` exits `0` passe
9. If readiness fails, run the automatic rollback (restore, reinstall, restart, poll).
10. Append the release record to `<stateDir>/releases/<service>.jsonl`.
++### Backups
++
++`sudo ovhost backup --all --offsite` runs daily from `deploy/systemd/openvibe-backup.timer` (03:30 UTC). It works like this:
++
++- It backs up every service that declares databases. A failing service never stops the others; exit `2` if anything failed.
++- It keeps the last 7 daily and 4 weekly good backups per service.
++- It writes a JSON summary to `<stateDir>/backup-runs/<run>.json`.
++- It encrypts every copy on the host (AES-256-GCM, Node crypto, with a key that stays on the host) and uploads it to S3-compatible storage (Backblaze B2). Off-host copies are kept for 30 days.
++- Backups are `root:root`: directories 0700, files 0600, no `-wal`/`-shm`.
++
++`ovhost restore-download` fetches, verifies and decrypts a run into a new root-only directory, and never touches a live database. See [docs/backups.md](docs/backups.md) for the install steps, the env names (`/etc/openvibe/backup.env`), the encryption format and the restore procedure.
++
### Restore drills (Wave 22)
`sudo ovhost drill <service>` restores the latest `ovhost backup` (or `--backup <dir>`) and starts a second instance on a spare loopback port. It compares that instance with production, stops it and logs the result to `<stateDir>/drills/<service>.jsonl`. It also prints a Markdown row for [docs/restore-drills.md](docs/restore-drills.md), which covers the steps, the sandbox and the per-service status.
@@@ -148,9 +172,10 @@ Nothing here has been done yet. These are the steps:
```
The tree must stay world-readable (default 0755/0644). The SQLite worker runs as `ubuntu` from this directory.
2. **Inventory:** `sudo install -o root -g root -m 0640 host.example.json /etc/openvibe/host.json`, then correct it against the machine. Check the Community database path, the Tools unit list, and Events, which was not deployed as of 22 Sep.
--3. **State:** `/var/lib/openvibe-host` (root, 0750) holds the release logs, locks, snapshots and drill logs. `/var/lib/openvibe-drills` (root, 0711) holds one directory per running drill, owned by the service user and removed afterwards. `/var/backups/openvibe/<service>` is created per service, owned by the service user, so the backup worker can write there. Both directories are created on first use.
++3. **State:** `/var/lib/openvibe-host` (root, 0750) holds the release logs, locks, snapshots, drill logs and backup run summaries. `/var/lib/openvibe-drills` (root, 0711) holds one directory per running drill, owned by the service user and removed afterwards. `/var/backups/openvibe` and everything in it is root-only (directories 0700, files 0600). The backup worker writes as the service user into its own directory under `/var/backups/openvibe.staging` (root, 0711), and root takes each copy over from there. All of these are created on first use.
4. **Run as root:** `sudo ovhost …`. `ovhost` drops to `ubuntu` for git/npm/SQLite (`runuser`) and needs root for `systemctl`, `nginx -t`, reading `/etc/openvibe/*.env` (names only) and writing vhosts. For least privilege, give operator accounts a sudoers rule for `/usr/local/bin/ovhost` alone instead of a general `ALL`. Do **not** grant it to the `ubuntu` service user if that account has no sudo today.
5. **First run:** `sudo ovhost status`, then `sudo ovhost validate <each service>`, then `sudo ovhost plan live`, all read-only apart from `git fetch`. Adopt `deploy` one service at a time, starting with one that has no protected sessions.
++6. **Scheduled backups:** see [docs/backups.md](docs/backups.md#installing-on-the-host-operator) (bucket, key, `/etc/openvibe/backup.env`, `openvibe-backup.timer`).
## Stage B: tenant static hosting
ddiff --git a/deploy/systemd/openvibe-backup.service b/deploy/systemd/openvibe-backup.service
nnew file mode 100644
iindex 0000000..2f2087c
---- /dev/null
++++ b/deploy/systemd/openvibe-backup.service
@@@ -0,0 +1,24 @@
++[Unit]
++Description=OpenVibe backups: every service's SQLite databases, then an encrypted off-host copy
++Documentation=https://github.com/OpenVibers/OpenVibe.Host/blob/main/docs/backups.md
++After=network-online.target
++Wants=network-online.target
++
++[Service]
++Type=oneshot
++# Root: ovhost drops to each service user for the SQLite backup (runuser) and is the only reader of
++# /etc/openvibe/backup.env, the encryption key and the root-only backups. The env file is read by
++# ovhost itself, not loaded into this unit's environment.
++User=root
++ExecStart=/usr/local/bin/ovhost backup --all --offsite
++Environment=PATH=/usr/local/bin:/usr/bin:/bin
++Environment=OVHOST_INVENTORY=/etc/openvibe/host.json
++UMask=0077
++Nice=10
++IOSchedulingClass=best-effort
++IOSchedulingPriority=7
++# media.db and live.db are the large ones; the SQLite worker's own limit is 6 h per database.
++TimeoutStartSec=6h
++PrivateTmp=true
++NoNewPrivileges=true
++ProtectSystem=full
ddiff --git a/deploy/systemd/openvibe-backup.timer b/deploy/systemd/openvibe-backup.timer
nnew file mode 100644
iindex 0000000..467920e
---- /dev/null
++++ b/deploy/systemd/openvibe-backup.timer
@@@ -0,0 +1,14 @@
++[Unit]
++Description=Daily OpenVibe backups (03:30 UTC, up to 15 min later)
++Documentation=https://github.com/OpenVibers/OpenVibe.Host/blob/main/docs/backups.md
++
++[Timer]
++OnCalendar=*-*-* 03:30:00 UTC
++RandomizedDelaySec=15min
++AccuracySec=1min
++# A run missed while the host was down happens at the next boot.
++Persistent=true
++Unit=openvibe-backup.service
++
++[Install]
++WantedBy=timers.target
ddiff --git a/docs/backups.md b/docs/backups.md
nnew file mode 100644
iindex 0000000..2740e71
---- /dev/null
++++ b/docs/backups.md
@@@ -0,0 +1,378 @@
++# Backups: scheduled, local and off-host
++
++`ovhost backup --all --offsite` runs once a day from a systemd timer. It backs up every database
++the inventory declares, keeps a week of daily and a month of weekly copies on the host, and uploads
++an encrypted copy of each run to S3-compatible object storage (Backblaze B2). This page covers what it
++does, how to install it, and how to restore from the off-host copy.
++
++Restore drills (a second instance started from a backup and compared with production) are in
++[restore-drills.md](restore-drills.md). A drill works on an off-host copy too: see
++[Testing a restore](#testing-a-restore).
++
++## What is backed up
++
++`ovhost backup --all` backs up every service in the inventory (`/etc/openvibe/host.json`) that has
++a `databases` list. With `host.example.json` that means:
++
++| Service | Databases |
++|---|---|
++| network | `network.db` |
++| live | `live.db` (includes the money ledger while `BILLING_AUTHORITY=live`), `rs-companion.db`. `data/analytics.db` is left out on purpose ([storage-state.md](storage-state.md): analytics, hard-coded path, not canonical). |
++| media | `media.db` (the media files themselves are in B2/R2, not covered here) |
++| tools | seven `apps/*/data/analytics.db`, and `jobs.db` for img, audio and docs |
++| games | `world.db` |
++| ai | `ai.db` |
++| openre | `openre.db` |
++| community, events, chat, billing, host, search, sources, wiki, blog, tips, vip, news, reviews, deals, coupons, trade, codes | one database each |
++
++A database that does not exist yet (a service that is not deployed) is reported as `skipped` and
++does not fail the run. Adding a database to the inventory is all it takes to include it.
++
++## One run
++
++For each service, one after another:
++
++1. **Copy.** The SQLite online backup API (`better-sqlite3` `.backup()`) makes a consistent copy while
++ the service keeps running. The copy is made by a worker running **as the service user**: root never
++ opens a service database, because a root-owned `-wal`/`-shm` file would stop the service from
++ writing its own database.
++2. **Check.** The worker switches the copy to `journal_mode=DELETE`, which checkpoints it and removes
++ its `-wal`/`-shm`, and runs `PRAGMA quick_check`. A copy that does not answer `ok` fails.
++3. **Take over.** The worker writes into `/var/backups/openvibe.staging/<service>-<stamp>/`, which
++ belongs to the service user. Root then takes that directory back (`root:root 0700`), so the service
++ user can no longer reach it. Root checks that each copy is a plain file with one link, and makes it
++ `root:root 0600`. It then renames the directory to `/var/backups/openvibe/<service>/<stamp>/`.
++4. **Record.** The backup is appended to `/var/lib/openvibe-host/backups/<service>.jsonl`, with
++ `ok: false` when any database failed.
++5. **Prune** (see [Retention](#retention)).
++
++**Permissions.** Backups hold user data. Every directory under `/var/backups/openvibe` is
++`root:root 0700` and every copy is `root:root 0600`, with no `-wal`/`-shm` next to it. Directories left
++by older versions (`ubuntu`, 0750) are locked down on the next run. The staging parent
++`/var/backups/openvibe.staging` is `root:root 0711`, and each run removes its own staging directory.
++
++**Failures.** A failing database never stops the other databases of the service, and a failing
++service never stops the other services. The run's exit code is `2` when anything failed and `0`
++otherwise. If a service fails halfway (one of two databases), the good copy is kept but the backup is
++marked `ok: false`. Such a backup is never picked by a drill, and it is pruned once a newer good backup
++exists.
++
++**Summary.** Each run writes `/var/lib/openvibe-host/backup-runs/<run>.json` (`root`, 0640). It holds
++per-service status, files, sizes, errors, what was pruned, and the off-host result: objects, bytes,
++failures, and the key id. The summary never holds a secret. The run id is the start time in UTC, for
++example `20260924-033412`.
++
++```
++sudo ovhost backup --all # local only
++sudo ovhost backup --all --offsite # what the timer runs
++sudo ovhost backup --all --json # the summary on stdout
++sudo ovhost backup --all --keep-daily 14 --keep-weekly 8
++sudo ovhost backup --all --no-prune
++```
++
++## Retention
++
++**On the host** (per service). Kept: the newest good backup of each of the last **7 days** that have
++one, and the newest good backup of each of the last **4 ISO weeks** that have one. Everything else is
++pruned, with three safeguards:
++
++- Only directories named by a stamp and recorded in the service's `.jsonl` are ever pruned. A
++ directory you copied in by hand is never touched.
++- Days and weeks without a good backup do not use up a slot. A service whose backups have been failing
++ for a month still keeps its last good ones.
++- A failed backup is removed only once a newer good one exists.
++
++**Off-host.** A run is deleted **30 days** after its run id (`BACKUP_OFFSITE_RETENTION_DAYS`) by
++`ovhost` itself. This only happens after a run that uploaded everything, and the newest 7 runs with a
++manifest are always kept. Failing uploads therefore never let the last good off-host copies age out.
++To leave deletion to a bucket lifecycle rule instead, set `BACKUP_OFFSITE_PRUNE=0` (see
++[Bucket](#bucket-backblaze-b2)).
++
++## Off-host copies
++
++### Layout
++
++```
++s3://<BACKUP_S3_BUCKET>/<BACKUP_S3_PREFIX>/<host>/<run>/<service>/<name>.db.ovbk
++s3://<BACKUP_S3_BUCKET>/<BACKUP_S3_PREFIX>/<host>/<run>/manifest.json
++```
++
++`<host>` is the inventory's `host` (`openvibe-oregon`), or the machine's hostname if the inventory has
++none.
++
++- Every `.ovbk` object is encrypted on the host before upload. Files are streamed:
++ - read,
++ - hashed,
++ - encrypted,
++ - uploaded, in one `PutObject` below 64 MiB, or as a multipart upload in 64 MiB parts.
++
++ Nothing extra is written to disk, and memory use is about one part.
++- `manifest.json` is uploaded last and is not encrypted. It holds:
++ - service and file names and source paths;
++ - plaintext and ciphertext sizes and SHA-256 hashes;
++ - the key id;
++ - any upload failures;
++ - an HMAC-SHA256 over the manifest, made with a key derived from the backup key.
++
++ It holds no data and no secret. `restore-download` refuses a manifest whose HMAC does not verify, and
++ a file whose hashes do not match it. This catches a swapped or rolled-back object, not only a
++ corrupted one.
++
++### Encryption format (OVBKAES1)
++
++Node's built-in `crypto` does the encryption: AES-256-GCM over 1 MiB chunks. The implementation is
++in `lib/backup-crypto.js`. Integers are big-endian.
++
++| Offset | Bytes | Field |
++|---|---|---|
++| 0 | 8 | magic `OVBKAES1` |
++| 8 | 4 | chunk size C (plaintext bytes per chunk; ovhost writes 1 MiB) |
++| 12 | 32 | salt, random per file |
++| 44 | 8 | key id: `HMAC-SHA256(master, "openvibe-backup key-id v1")`, first 8 bytes |
++| 52 | … | chunks: each is ciphertext (C bytes; the last one 0..C) followed by its 16-byte GCM tag |
++
++- The file key is `HKDF-SHA256(master, salt, "openvibe-backup v1 file key")`, 32 bytes.
++- The nonce of chunk *i* is 11 bytes of *i* (big-endian), then one flag byte: `1` on the last chunk,
++ `0` on the others.
++- Every chunk's AAD is the 52 header bytes.
++- The last chunk is always present, even when it is empty.
++
++Because of this format:
++- a changed byte fails authentication;
++- truncation fails, because the last remaining chunk lacks the final flag;
++- appended data fails;
++- an edited header fails;
++- a file encrypted with another key is reported by its key id.
++
++The manifest HMAC key is `HKDF-SHA256(master, "", "openvibe-backup v1 manifest")`.
++
++### The key
++
++The encryption key is 32 random bytes in a root-only file on the host. ovhost refuses the key file,
++and the env file, unless they are owned by root and not readable by group or others.
++
++**The key must also exist somewhere other than this host.** If the host is lost, the key is lost
++with it, and every off-host copy becomes unreadable. Keep one copy offline, for example in the owner's
++password manager, or on paper in a safe. Never put a copy in the bucket, and never put one in a
++repository. `ovhost offsite check` prints the key id. Write that id next to the escrowed copy, so you
++can match the copy to its backups later.
++
++## Installing on the host (operator)
++
++Run these as root on the host. They install nothing system-wide beyond what ovhost already has:
++the S3 client is an npm dependency of OpenVibe.Host (`@aws-sdk/client-s3`, the same client Media uses).
++
++1. **Update ovhost** (from the Host README's install layout):
++ ```
++ cd /usr/local/lib/openvibe-host
++ sudo git pull --ff-only
++ sudo npm ci --omit=dev --no-audit --no-fund
++ sudo git checkout -- package-lock.json # only if npm rewrote it (npm 9 on the host)
++ ovhost --help | grep -q 'backup --all' && echo ok
++ ```
++2. **Bucket and key** (Backblaze web UI or `b2` CLI; see [Bucket](#bucket-backblaze-b2)):
++ - a private bucket, for example `openvibe-backups` (not Media's bucket);
++ - an application key restricted to that bucket.
++3. **Encryption key**, generated on the host and escrowed off it:
++ ```
++ sudo sh -c 'umask 077; openssl rand -hex 32 > /etc/openvibe/backup.key'
++ sudo chown root:root /etc/openvibe/backup.key && sudo chmod 0600 /etc/openvibe/backup.key
++ sudo cat /etc/openvibe/backup.key # copy it into the password manager now, then clear the screen
++ ```
++4. **Env file** `/etc/openvibe/backup.env` (root:root 0600):
++ ```
++ sudo install -o root -g root -m 0600 /dev/null /etc/openvibe/backup.env
++ sudoedit /etc/openvibe/backup.env
++ ```
++ ```
++ BACKUP_S3_ENDPOINT=https://s3.us-west-004.backblazeb2.com
++ BACKUP_S3_BUCKET=openvibe-backups
++ BACKUP_S3_PREFIX=openvibe-backups
++ BACKUP_S3_KEY_ID=<application key id>
++ BACKUP_S3_SECRET=<application key>
++ BACKUP_ENCRYPTION_KEY_FILE=/etc/openvibe/backup.key
++ # optional
++ # BACKUP_S3_REGION=us-west-004 (derived from a B2 endpoint; else us-east-1)
++ # BACKUP_S3_FORCE_PATH_STYLE=1 (default 1; B2 wants path-style)
++ # BACKUP_OFFSITE_RETENTION_DAYS=30
++ # BACKUP_OFFSITE_PRUNE=1 (0 when a bucket lifecycle rule deletes old runs)
++ ```
++ Use the endpoint and region of the bucket's own B2 region. Media's `MEDIA_B2_ENDPOINT` shows the
++ format. Media's bucket and key are separate, and are not reused here.
++5. **Check** that the config, the key and the bucket work, and do a first run by hand:
++ ```
++ sudo ovhost offsite check
++ sudo ovhost backup --all --offsite # exit 0; prints one line per service and the OFFSITE line
++ sudo ovhost offsite list
++ ```
++6. **Timer:**
++ ```
++ sudo install -o root -g root -m 0644 deploy/systemd/openvibe-backup.service deploy/systemd/openvibe-backup.timer /etc/systemd/system/
++ sudo systemctl daemon-reload
++ sudo systemctl enable --now openvibe-backup.timer
++ systemctl list-timers openvibe-backup.timer
++ ```
++ (Run these from `/usr/local/lib/openvibe-host`.) The timer fires daily at 03:30 UTC, up to 15 minutes
++ later (`RandomizedDelaySec`). A run missed while the host was down happens at the next boot
++ (`Persistent=true`). To run one now: `sudo systemctl start openvibe-backup.service`.
++7. **Disk.** Local retention keeps up to 11 copies of every database. Check the space before the first
++ week fills up: `sudo du -sh /var/backups/openvibe/*` against `df -h /var/backups`. If space is tight,
++ lower `--keep-daily`/`--keep-weekly` in the unit's `ExecStart`.
++
++### Watching it
++
++```
++systemctl status openvibe-backup.service # last result: exit 0, or status=2/INVALIDARGUMENT on a failure
++journalctl -u openvibe-backup.service --since today
++sudo ls -t /var/lib/openvibe-host/backup-runs | head -1
++sudo jq '{ok, services: [.services[] | {service, status, error}], offsite: {ok: .offsite.ok, failures: .offsite.failures}}' /var/lib/openvibe-host/backup-runs/<run>.json
++```
++
++Nothing alerts on a failed run yet. The unit fails (exit 2) and the summary says `"ok": false`. Hook
++an `OnFailure=` unit or the monitoring stack to that when one exists.
++
++### Bucket (Backblaze B2)
++
++- **Private** bucket, used for backups only, in a different account or bucket from Media's content.
++- **Application key**, restricted to this bucket. It needs `listFiles`, `readFiles` and `writeFiles`,
++ plus `deleteFiles` when ovhost prunes (`BACKUP_OFFSITE_PRUNE=1`, the default).
++- **Lifecycle.** B2 keeps old versions by default: a delete through the S3 API only *hides* the file.
++ Add a lifecycle rule on the prefix with `daysFromHidingToDeleting: 1`, so pruned runs really go. In
++ the UI this is "Keep only the last version of the file".
++- **Stronger option** against a compromised host deleting its own backups:
++ - give the key no `deleteFiles`;
++ - set `BACKUP_OFFSITE_PRUNE=0`;
++ - let the lifecycle rule delete old runs: `daysFromUploadingToHiding: 30`,
++ `daysFromHidingToDeleting: 1`.
++
++ You can also enable Object Lock on the bucket with a 30-day retention. The trade-off is that the
++ "always keep the newest 7 runs" safeguard no longer applies, because the bucket deletes by age alone.
++
++## Restoring from an off-host copy
++
++`ovhost restore-download` only downloads, verifies and decrypts. **It never writes into a live
++database, a service checkout or a database directory**, and it never overwrites anything:
++- the target directory must not exist, or must be empty;
++- the target must not overlap any declared database directory, checkout, `/var/backups/openvibe` or
++ `/var/lib/openvibe-host`;
++- every file is created exclusively.
++
++Putting a copy in place is a separate, manual step.
++
++### 1. Find the run
++
++```
++sudo ovhost offsite list # every run, newest first
++sudo ovhost offsite list live # runs that include live
++```
++
++### 2. Download, verify and decrypt
++
++```
++sudo ovhost restore-download live latest
++sudo ovhost restore-download live 20260924-033412
++sudo ovhost restore-download live 20260924-033412 --out /var/lib/openvibe-restore/live-before-incident
++```
++
++The default target is `/var/lib/openvibe-restore/<service>-<run>/` (`root:root 0700`, files `0600`).
++For each file, the command:
++
++- checks the manifest's HMAC;
++- streams the object through GCM decryption;
++- checks both SHA-256 hashes against the manifest;
++- runs `PRAGMA quick_check` on the result.
++
++Any failure removes that file and exits non-zero. The output lists each file with its size, its hash,
++`quick_check ok` and the production path it came from.
++
++### 3. Put it in place (manual, one service at a time)
++
++Take a fresh local backup of the current state first, even a broken one, so the restore can be undone.
++Then stop the service, move its current database aside, install the copy as the service user's file,
++and start the service again. For Live, check protected sessions first (`ovhost status live`). The
++socket unit stays up, and only the service unit stops.
++
++```
++sudo ovhost backup live # the state you are about to replace
++sudo systemctl stop openvibe-live.service # never the .socket unit
++cd /opt/openvibe.live/data
++sudo mv live.db live.db.replaced-$(date -u +%Y%m%d-%H%M%S)
++sudo rm -f live.db-wal live.db-shm # only after the service has stopped
++sudo install -o ubuntu -g ubuntu -m 0640 /var/lib/openvibe-restore/live-20260924-033412/live.db live.db
++sudo systemctl start openvibe-live.service
++curl -s 127.0.0.1:3000/api/ready | jq .status
++```
++
++- Use the production path from the restore output (`was …`). The owner and mode must match the other
++ files in that directory (`ls -l`).
++- For services with a `StateDirectory` (`/var/lib/openvibe-*`), the directory is owned by the service
++ user.
++- Remove `/var/lib/openvibe-restore/<service>-<run>` when you are done: it holds user data.
++- **Money.** `live.db` holds the ledger while `BILLING_AUTHORITY=live`. A restore loses every
++ transaction made after the backup. Before any money action resumes, reconcile against the payment
++ provider for the gap (see [cutover-runbook.md](cutover-runbook.md)). Consider setting the owner-only
++ `money_writes_frozen` flag first.
++
++### Disaster recovery: a new host
++
++The bucket, the escrowed key and this repository are enough.
++
++1. Install ovhost as in the Host README.
++2. Copy `host.example.json` to `/etc/openvibe/host.json` (root:root 0640).
++3. Write `/etc/openvibe/backup.key` from the escrowed copy (root:root 0600), and check that
++ `sudo ovhost offsite check` prints the same key id.
++4. Write `/etc/openvibe/backup.env`.
++5. The new machine has another hostname, so name the old host explicitly:
++ ```
++ sudo ovhost offsite list --from-host openvibe-oregon
++ sudo ovhost restore-download live latest --from-host openvibe-oregon
++ ```
++6. Deploy each service's checkout and unit, and put each database in place as in step 3, before its
++ first start.
++
++## Testing a restore
++
++A restore is proven only when a restored copy runs. The downloaded directory has the same file names
++as a local backup, so a restore drill can use it directly (for services with a drill block):
++
++```
++sudo ovhost backup --all --offsite # a fresh run, so counts match production
++sudo ovhost restore-download community latest --out /var/lib/openvibe-restore/drill-community
++sudo ovhost drill community --backup /var/lib/openvibe-restore/drill-community
++sudo rm -rf /var/lib/openvibe-restore/drill-community
++```
++
++The drill:
++- copies the databases into its sandbox as the service user (`install -o <user> -m 0600`);
++- runs `integrity_check`;
++- starts a sandboxed second instance and compares it with production;
++- stops it and cleans up.
++
++Record the result in [restore-drills.md](restore-drills.md) and note "off-host" in the Backup column.
++
++For services without a drill (live, media, tools, games, openre), download the copy and check it by
++hand:
++
++```
++sudo ovhost restore-download live latest --out /var/lib/openvibe-restore/check-live
++sudo sqlite3 -readonly /var/lib/openvibe-restore/check-live/live.db 'PRAGMA integrity_check; SELECT count(*) FROM users;'
++sudo rm -rf /var/lib/openvibe-restore/check-live
++```
++
++(`sqlite3` is the Debian package of the same name. `restore-download` has already run `quick_check`;
++this adds a full `integrity_check` and a look at the data.)
++
++Do this at least once after installing, and again after any change to the key or the bucket.
++
++## Commands
++
++| Command | What it does | Exit |
++|---|---|---|
++| `ovhost backup --all [--offsite] [--no-prune] [--keep-daily n] [--keep-weekly n]` | back up every service with databases; prune; summary; optionally upload | 0 all ok · 1 lock/usage · 2 any failure |
++| `ovhost offsite push [--run <run>]` | upload a run (default: the latest) again, for example after a failed upload | 0 · 1 config · 2 failure |
++| `ovhost offsite list [<service>] [--from-host <h>]` | runs off-host, newest first | 0 · 1 config · 2 bucket error |
++| `ovhost offsite check` | config, key and bucket access; uploads nothing | 0 · 1 config · 2 bucket error |
++| `ovhost restore-download <service> <run\|latest> [--out <dir>] [--from-host <h>]` | download, verify and decrypt into a new root-only directory | 0 · 1 refused · 2 verification failed |
++
++The offsite commands and `restore-download` need root. They take `--backup-env <file>`, whose default
++is `/etc/openvibe/backup.env` (or `$OVHOST_BACKUP_ENV`).
ddiff --git a/docs/restore-drills.md b/docs/restore-drills.md
iindex 0ba1069..3fb2fc4 100644
---- a/docs/restore-drills.md
++++ b/docs/restore-drills.md
@@@ -42,6 +42,20 @@ instance could also still reach production Network over loopback (JWKS, service
resolve-batch). The inventory's `community` drill block now also empties `OV_OAUTH_CLIENT_SECRET` and
points `OV_LIVE_INTERNAL_URL` and `OV_MEDIA_INTERNAL_URL` at a closed port.
++## Drilling an off-host copy
++
++Scheduled backups and their encrypted off-host copies are described in [backups.md](backups.md).
++To drill a copy from the bucket, not the local disk, download it first. The directory
++`restore-download` writes has the layout `--backup` expects:
++
++```
++sudo ovhost restore-download community latest --out /var/lib/openvibe-restore/drill-community
++sudo ovhost drill community --backup /var/lib/openvibe-restore/drill-community
++sudo rm -rf /var/lib/openvibe-restore/drill-community
++```
++
++Write "off-host" and the run id in the Backup column of the table above.
++
## Running a drill with `ovhost drill`
```
@@@ -57,7 +71,9 @@ block (see `host.example.json`):
1. **Restore.** The command picks the latest `ovhost backup` from `<stateDir>/backups/<service>.jsonl`,
or the `--backup` directory. It copies every database named in `drill.databases` into
`/var/lib/openvibe-drills/<service>-<stamp>/db/`, a directory owned by the service user (mode 0700).
-- The copy is made as the service user. `PRAGMA integrity_check` must return `ok` for each copy;
++ Backups are root-only (directories 0700, files 0600, see [backups.md](backups.md)), so root makes
++ the copy with `install -o <service user> -m 0600`, and the copy belongs to the service user.
++ `PRAGMA integrity_check`, run as the service user, must return `ok` for each copy;
otherwise the drill fails before anything starts.
2. **Start.** A second instance starts from the production checkout through `systemd-run`, as the
service user. It uses the production unit's `ExecStart` and `WorkingDirectory`, the production
ddiff --git a/docs/storage-state.md b/docs/storage-state.md
iindex 88210db..dd58584 100644
---- a/docs/storage-state.md
++++ b/docs/storage-state.md
@@@ -59,7 +59,9 @@ Sources:
- Games `/healthz` (the inventory now uses `/api/ready`, 2026-09-23; Network, Media and Tools too).
The inventory should point at `/api/ready` where one exists. Games has none.
--4. **Two services have no ovhost inventory entry, so no `ovhost backup` and no drill**: AI (`ai.db`) and
++4. *(Fixed: the inventory has had AI and OpenRe entries since `f0dbd0c`, and both are in the scheduled
++ `ovhost backup --all` with off-host copies, [backups.md](backups.md). Their drills are still unsupported.)*
++ **Two services have no ovhost inventory entry, so no `ovhost backup` and no drill**: AI (`ai.db`) and
OpenRe.Stream (`openre.db`). Whether they are backed up any other way is unknown. Check the host's cron
and timers.
5. **The inventory's Tools backup list is incomplete.** It names the seven `analytics.db` files only.
ddiff --git a/host.example.json b/host.example.json
iindex 1f0c269..95dae7b 100644
---- a/host.example.json
++++ b/host.example.json
@@@ -3,6 +3,7 @@
"host": "openvibe-oregon",
"stateDir": "/var/lib/openvibe-host",
"backupDir": "/var/backups/openvibe",
++ "backupStagingDir": "/var/backups/openvibe.staging",
"drillDir": "/var/lib/openvibe-drills",
"_drill": "Restore drills (ovhost drill <service>): each service with a database has a drill block. The drill port is the service port + 10000 on 127.0.0.1. http://127.0.0.1:9 is a closed port, used where an empty value would fall back to a production service on loopback. An empty OV_OAUTH_CLIENT_SECRET stops a service from minting service tokens, so it cannot call other services with its identity. Loopback stays reachable from the drill sandbox (services need Network's public keys), so these overrides are what keep a drill away from production. _notes explain what is left on and why.",
"nginx": {
ddiff --git a/lib/backup-crypto.js b/lib/backup-crypto.js
nnew file mode 100644
iindex 0000000..0852846
---- /dev/null
++++ b/lib/backup-crypto.js
@@@ -0,0 +1,197 @@
++'use strict';
++/**
++ * Encryption for off-host backups: AES-256-GCM in fixed-size chunks, with Node crypto only.
++ *
++ * The master key is 32 random bytes kept in a root-only file on the host (64 hex characters, or
++ * base64). It never leaves the host and is never printed; `keyId()` is a non-secret fingerprint
++ * that names it in manifests and logs.
++ *
++ * File format OVBKAES1 (integers big-endian; also documented in docs/backups.md):
++ *
++ * header 52 bytes
++ * 0 8 magic "OVBKAES1"
++ * 8 4 chunk size C (plaintext bytes per chunk; ovhost writes 1 MiB, readers accept up to 64 MiB)
++ * 12 32 salt: random per file
++ * 44 8 key id: HMAC-SHA256(master, "openvibe-backup key-id v1")[0..8]
++ * chunks each: ciphertext (C bytes; the final chunk 0..C bytes) || 16-byte GCM tag
++ *
++ * file key = HKDF-SHA256(master, salt, info "openvibe-backup v1 file key", 32 bytes)
++ * nonce_i = 11-byte big-endian chunk index i || 1 byte final flag (1 on the last chunk, else 0)
++ * AAD = the 52 header bytes, for every chunk
++ *
++ * A fresh salt gives every file its own key, so the counter nonces never repeat under one key.
++ * The final flag makes truncation at a chunk boundary detectable, and the header in the AAD makes
++ * any edit to it (chunk size, salt, key id) fail authentication. The final chunk is always written,
++ * even when it is empty, so a file always ends with a flagged chunk.
++ */
++const crypto = require('crypto');
++const { Transform } = require('stream');
++
++const MAGIC = Buffer.from('OVBKAES1', 'ascii');
++const HEADER_LEN = 52;
++const TAG_LEN = 16;
++const DEFAULT_CHUNK = 1024 * 1024;
++const MAX_CHUNK = 64 * 1024 * 1024;
++
++class BackupCryptoError extends Error {}
++
++/** Parse a key file's text: 64 hex characters or base64 of 32 bytes. The error never echoes it. */
++function parseKey(text) {
++ const t = String(text == null ? '' : text).trim();
++ if (/^[0-9a-fA-F]{64}$/.test(t)) return Buffer.from(t, 'hex');
++ if (/^[A-Za-z0-9+/]{43}=?$/.test(t)) {
++ const b = Buffer.from(t, 'base64');
++ if (b.length === 32) return b;
++ }
++ throw new BackupCryptoError('the encryption key file must hold 32 random bytes as 64 hex characters (openssl rand -hex 32)');
++}
++
++function keyId(master) {
++ return crypto.createHmac('sha256', master).update('openvibe-backup key-id v1').digest().subarray(0, 8).toString('hex');
++}
++
++function fileKey(master, salt) {
++ return Buffer.from(crypto.hkdfSync('sha256', master, salt, 'openvibe-backup v1 file key', 32));
++}
++
++/** A key for authenticating run manifests, separate from every file key. */
++function manifestKey(master) {
++ return Buffer.from(crypto.hkdfSync('sha256', master, Buffer.alloc(0), 'openvibe-backup v1 manifest', 32));
++}
++
++function nonce(index, final) {
++ const n = Buffer.alloc(12);
++ n.writeBigUInt64BE(BigInt(index), 3);
++ n[11] = final ? 1 : 0;
++ return n;
++}
++
++function header(master, chunkSize, salt) {
++ const h = Buffer.alloc(HEADER_LEN);
++ MAGIC.copy(h, 0);
++ h.writeUInt32BE(chunkSize, 8);
++ salt.copy(h, 12);
++ Buffer.from(keyId(master), 'hex').copy(h, 44);
++ return h;
++}
++
++/** Ciphertext size for a plaintext of `plainBytes`. */
++function encryptedSize(plainBytes, chunkSize = DEFAULT_CHUNK) {
++ const chunks = Math.max(1, Math.ceil(plainBytes / chunkSize));
++ return HEADER_LEN + plainBytes + chunks * TAG_LEN;
++}
++
++function sealChunk(key, aad, index, final, plain) {
++ const c = crypto.createCipheriv('aes-256-gcm', key, nonce(index, final));
++ c.setAAD(aad);
++ return Buffer.concat([c.update(plain), c.final(), c.getAuthTag()]);
++}
++
++function openChunk(key, aad, index, final, sealed) {
++ const d = crypto.createDecipheriv('aes-256-gcm', key, nonce(index, final));
++ d.setAAD(aad);
++ d.setAuthTag(sealed.subarray(sealed.length - TAG_LEN));
++ try {
++ return Buffer.concat([d.update(sealed.subarray(0, sealed.length - TAG_LEN)), d.final()]);
++ } catch {
++ throw new BackupCryptoError(`chunk ${index} failed authentication: the file was altered, truncated or encrypted with another key`);
++ }
++}
++
++function createEncryptStream(master, { chunkSize = DEFAULT_CHUNK, salt = crypto.randomBytes(32) } = {}) {
++ if (!Buffer.isBuffer(master) || master.length !== 32) throw new BackupCryptoError('the master key must be 32 bytes');
++ if (!(Number.isInteger(chunkSize) && chunkSize >= 1 && chunkSize <= MAX_CHUNK)) throw new BackupCryptoError('bad chunk size');
++ const aad = header(master, chunkSize, salt);
++ const key = fileKey(master, salt);
++ let buf = Buffer.alloc(0);
++ let index = 0;
++ let started = false;
++ return new Transform({
++ transform(chunk, _enc, cb) {
++ if (!started) { this.push(aad); started = true; }
++ buf = buf.length ? Buffer.concat([buf, chunk]) : Buffer.from(chunk);
++ // Only emit a chunk when more data follows it, so the last one can carry the final flag.
++ while (buf.length > chunkSize) {
++ this.push(sealChunk(key, aad, index++, false, buf.subarray(0, chunkSize)));
++ buf = buf.subarray(chunkSize);
++ }
++ cb();
++ },
++ flush(cb) {
++ if (!started) this.push(aad);
++ this.push(sealChunk(key, aad, index++, true, buf));
++ buf = Buffer.alloc(0);
++ cb();
++ },
++ });
++}
++
++function createDecryptStream(master) {
++ if (!Buffer.isBuffer(master) || master.length !== 32) throw new BackupCryptoError('the master key must be 32 bytes');
++ let buf = Buffer.alloc(0);
++ let aad = null;
++ let key = null;
++ let sealedLen = 0;
++ let index = 0;
++ function readHeader() {
++ const h = buf.subarray(0, HEADER_LEN);
++ if (!h.subarray(0, 8).equals(MAGIC)) throw new BackupCryptoError('not an OpenVibe backup file (bad magic)');
++ const chunkSize = h.readUInt32BE(8);
++ if (!(chunkSize >= 1 && chunkSize <= MAX_CHUNK)) throw new BackupCryptoError('bad chunk size in header');
++ if (h.subarray(44, 52).toString('hex') !== keyId(master)) throw new BackupCryptoError(`encrypted with another key (key id ${h.subarray(44, 52).toString('hex')}, this host's key is ${keyId(master)})`);
++ aad = Buffer.from(h);
++ key = fileKey(master, h.subarray(12, 44));
++ sealedLen = chunkSize + TAG_LEN;
++ buf = buf.subarray(HEADER_LEN);
++ }
++ return new Transform({
++ transform(chunk, _enc, cb) {
++ try {
++ buf = buf.length ? Buffer.concat([buf, chunk]) : Buffer.from(chunk);
++ if (!aad) { if (buf.length < HEADER_LEN) return cb(); readHeader(); }
++ while (buf.length > sealedLen) {
++ this.push(openChunk(key, aad, index++, false, buf.subarray(0, sealedLen)));
++ buf = buf.subarray(sealedLen);
++ }
++ cb();
++ } catch (err) { cb(err); }
++ },
++ flush(cb) {
++ try {
++ if (!aad) throw new BackupCryptoError('truncated: no complete header');
++ if (buf.length < TAG_LEN) throw new BackupCryptoError('truncated: the final chunk is missing');
++ this.push(openChunk(key, aad, index++, true, buf));
++ cb();
++ } catch (err) { cb(err); }
++ },
++ });
++}
++
++/** A pass-through that hashes (SHA-256) and counts what flows through it. */
++function createDigestTap() {
++ const hash = crypto.createHash('sha256');
++ let bytes = 0;
++ const t = new Transform({
++ transform(chunk, _enc, cb) { hash.update(chunk); bytes += chunk.length; cb(null, chunk); },
++ });
++ t.result = () => ({ sha256: hash.digest('hex'), bytes });
++ return t;
++}
++
++function hmacManifest(master, body) {
++ return crypto.createHmac('sha256', manifestKey(master)).update(body).digest('hex');
++}
++
++module.exports = {
++ parseKey,
++ keyId,
++ createEncryptStream,
++ createDecryptStream,
++ createDigestTap,
++ encryptedSize,
++ hmacManifest,
++ BackupCryptoError,
++ HEADER_LEN,
++ TAG_LEN,
++ DEFAULT_CHUNK,
++};
ddiff --git a/lib/cli.js b/lib/cli.js
iindex 97f83ec..8007679 100644
---- a/lib/cli.js
++++ b/lib/cli.js
@@@ -30,6 +30,20 @@ Usage: ovhost <command> [args] [--inventory <file>] [--json]
database); lists domains still waiting for a certificate
snapshot <service> [--out <file>] config tarball: units, vhost, env NAMES, sha — no secret values
backup <service> sqlite .backup of the declared databases to a dated directory
++ backup --all [--offsite] [--no-prune] [--keep-daily <n>] [--keep-weekly <n>]
++ every service with databases; one failure never stops the
++ others; keeps the last 7 daily + 4 weekly good backups per
++ service; JSON summary in <stateDir>/backup-runs/<run>.json;
++ --offsite then encrypts and uploads the run (root)
++ offsite push [--run <run>] encrypt + upload a backup --all run (default: the latest)
++ offsite list [<service>] [--from-host <name>]
++ runs stored off-host, newest first
++ offsite check config, key and bucket access; uploads nothing
++ restore-download <service> <run|latest> [--out <dir>] [--from-host <name>]
++ download, verify and decrypt one service's databases into a
++ NEW root-only directory; never touches a live database
++ (offsite commands: --backup-env <file>, default
++ /etc/openvibe/backup.env)
drill <service> [--backup <dir>] [--keep]
restore drill (root): restore the latest backup into a temp
dir, start a sandboxed second instance on the drill port, compare
@@@ -38,10 +52,11 @@ Usage: ovhost <command> [args] [--inventory <file>] [--json]
Inventory: --inventory, else $OVHOST_INVENTORY, else /etc/openvibe/host.json.
Exit codes: 0 ok · 1 usage/precondition · 2 validation failed (nothing restarted) · 3 not ready,
rolled back · 4 rollback failed (manual intervention) · 5 protected sessions active ·
--drill: 0 passed · 1 precondition (not root, port in use, no backup, unsupported) · 2 drill failed
++drill: 0 passed · 1 precondition (not root, port in use, no backup, unsupported) · 2 drill failed ·
++backup --all / offsite: 0 everything succeeded · 1 precondition (config, key, lock) · 2 something failed
`;
--const FLAGS_WITH_VALUE = new Set(['inventory', 'to', 'limit', 'warn-days', 'variant', 'manifest', 'out', 'ready-timeout', 'wildcard-cert', 'backup']);
++const FLAGS_WITH_VALUE = new Set(['inventory', 'to', 'limit', 'warn-days', 'variant', 'manifest', 'out', 'ready-timeout', 'wildcard-cert', 'backup', 'run', 'keep-daily', 'keep-weekly', 'from-host', 'backup-env']);
function parseArgs(argv) {
const pos = [];
@@@ -68,7 +83,23 @@ const ICON = { ok: ' ok ', warn: 'WARN ', error: 'FAIL ', info: 'info ' };
function short(sha) { return sha ? sha.slice(0, 12) : '-'; }
--async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => process.stdout.write(`${s}\n`), env = process.env } = {}) {
++function count(opts, name, dflt) {
++ if (opts[name] == null) return dflt;
++ const n = Number(opts[name]);
++ if (!(Number.isInteger(n) && n >= 1 && n <= 366)) throw new UsageError(`--${name} must be a whole number from 1`);
++ return n;
++}
++
++function bytes(n) {
++ if (n == null) return '?';
++ const u = ['B', 'KB', 'MB', 'GB', 'TB'];
++ let i = 0;
++ let v = n;
++ while (v >= 1024 && i < u.length - 1) { v /= 1024; i += 1; }
++ return `${i ? v.toFixed(1) : v} ${u[i]}`;
++}
++
++async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => process.stdout.write(`${s}\n`), env = process.env, s3 = lib.offsite.createClient } = {}) {
let parsed;
try { parsed = parseArgs(argv); } catch (err) { out(err.message); out(USAGE); return 1; }
const { pos, opts } = parsed;
@@@ -200,10 +231,73 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc
return 0;
}
case 'backup': {
++ if (opts.all) {
++ if (pos[1]) throw new UsageError('backup --all takes no <service>');
++ const offsiteRun = opts.offsite ? async (summary) => {
++ const cfg = await lib.offsite.loadConfig(exec, { file: opts['backup-env'], env });
++ return lib.offsite.push(ctx, summary, { cfg, client: s3(cfg) });
++ } : null;
++ const r = await lib.backupAll(ctx, { daily: count(opts, 'keep-daily', 7), weekly: count(opts, 'keep-weekly', 4), prune: !opts['no-prune'], offsite: offsiteRun });
++ if (json) emit(r);
++ else {
++ for (const s of r.services) {
++ const copied = s.files.filter((f) => f.dest && !f.error);
++ const size = copied.reduce((a, f) => a + (f.bytes || 0), 0);
++ const detail = s.status === 'failed' ? s.error : s.status === 'skipped' ? 'no database found' : `${copied.length} file(s), ${bytes(size)}`;
++ out(`${s.status.toUpperCase().padEnd(8)} ${s.service.padEnd(10)} ${detail}${s.pruned.length ? `; pruned ${s.pruned.length}` : ''}`);
++ }
++ if (r.offsite) {
++ const o = r.offsite;
++ if (o.error) out(`OFFSITE FAILED: ${o.error}`);
++ else out(`OFFSITE ${o.ok ? 'ok' : 'FAILED'}: ${o.uploaded.length} file(s) to s3://${o.bucket}/${o.location} (key ${o.keyId})${o.failures.length ? `; ${o.failures.length} failure(s): ${o.failures.map((f) => `${f.service ? `${f.service}/` : ''}${f.file}: ${f.error}`).join('; ')}` : ''}${o.pruned.length ? `; pruned ${o.pruned.length} old run(s)` : ''}`);
++ }
++ out(`run ${r.run}: ${r.ok ? 'ok' : 'FAILED'}; summary ${r.file}`);
++ }
++ return r.ok ? 0 : 2;
++ }
const r = await lib.backup(ctx, needService());
if (json) emit(r); else for (const f of r.files) out(f.skipped ? `skipped ${f.source}: ${f.skipped}` : `${f.source} → ${f.dest} (${f.bytes} bytes)`);
return 0;
}
++ case 'offsite': {
++ const sub = pos[1];
++ if (!['push', 'list', 'check'].includes(sub)) throw new UsageError('usage: ovhost offsite push [--run <run>] | offsite list [<service>] | offsite check');
++ const cfg = await lib.offsite.loadConfig(exec, { file: opts['backup-env'], env });
++ const client = s3(cfg);
++ if (sub === 'check') {
++ const r = await lib.offsite.check(ctx, { cfg, client });
++ if (json) emit(r); else out(`off-host ok: s3://${r.bucket}/${r.prefix}/${r.host}/ at ${r.endpoint} (region ${r.region}); key id ${r.keyId}; retention ${r.retentionDays} days${r.prune ? '' : ' (pruning off)'}`);
++ return 0;
++ }
++ if (sub === 'list') {
++ const host = await lib.offsite.hostName(exec, inv, opts['from-host']);
++ const runs = (await lib.offsite.listRuns(client, cfg, host)).filter((x) => !pos[2] || x.services.includes(pos[2])).map(({ keys, ...x }) => x);
++ if (json) { emit(runs); return 0; }
++ if (!runs.length) out(`no runs under s3://${cfg.bucket}/${cfg.prefix}/${host}/`);
++ for (const x of runs) out(`${x.run} ${String(x.objects).padStart(3)} object(s) ${bytes(x.bytes).padStart(9)} ${x.manifest ? '' : '(no manifest) '}${x.services.join(' ')}`);
++ return 0;
++ }
++ const summary = await lib.readRun(exec, inv, opts.run || null);
++ if (!summary) throw new UsageError(opts.run ? `no backup --all run ${opts.run} in ${inv.stateDir}/backup-runs` : `no backup --all run recorded in ${inv.stateDir}/backup-runs; run ovhost backup --all first`);
++ summary.offsite = await lib.offsite.push(ctx, summary, { cfg, client });
++ if (!summary.offsite.ok) summary.ok = false;
++ await lib.writeRun(exec, summary);
++ const o = summary.offsite;
++ if (json) emit(o);
++ else out(`OFFSITE ${o.ok ? 'ok' : 'FAILED'}: ${o.uploaded.length} file(s) to s3://${o.bucket}/${o.location}${o.failures.length ? `; failures: ${o.failures.map((f) => `${f.service ? `${f.service}/` : ''}${f.file}: ${f.error}`).join('; ')}` : ''}`);
++ return o.ok ? 0 : 2;
++ }
++ case 'restore-download': {
++ const id = needService();
++ if (!pos[2]) throw new UsageError('restore-download needs a <run> (a run id such as 20260923-033000, or latest)');
++ const cfg = await lib.offsite.loadConfig(exec, { file: opts['backup-env'], env });
++ const r = await lib.offsite.restoreDownload(ctx, id, pos[2], { cfg, client: s3(cfg), out: opts.out, host: opts['from-host'] });
++ if (json) { emit(r); return 0; }
++ for (const f of r.files) out(`${f.file} ${bytes(f.bytes)} sha256 ${f.sha256.slice(0, 16)}… quick_check ${f.quickCheck} (was ${f.source})`);
++ out(`restored ${r.service} from ${r.host}/${r.run} into ${r.dir} (root-only). Nothing live was touched;`);
++ out('see docs/backups.md "Restoring from an off-host copy" to put a copy in place.');
++ return 0;
++ }
case 'drill': {
const r = await lib.drill(ctx, needService(), { backup: opts.backup, keep: !!opts.keep });
if (json) emit(r.record);
@@@ -222,6 +316,7 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc
if (err instanceof InventoryError || err instanceof LockedError) { out(`[ovhost] ${err.message}`); return 1; }
if (err instanceof SocketGuardError) { out(`[ovhost] ${err.message}`); return 1; }
if (err instanceof lib.DrillError) { out(`[ovhost] ${err.message}`); return err.exitCode; }
++ if (err instanceof lib.offsite.OffsiteError) { out(`[ovhost] ✗ ${err.message}`); return err.exitCode; }
if (err instanceof lib.OpError) { out(`[ovhost] ✗ ${err.message}`); return err.exitCode; }
out(`[ovhost] ✗ ${err.message}`);
return 2;
ddiff --git a/lib/commands/backup.js b/lib/commands/backup.js
iindex a6c7bb6..6dfecbb 100644
---- a/lib/commands/backup.js
++++ b/lib/commands/backup.js
@@@ -5,40 +5,230 @@
* running as the service user (root never opens a service database — see sqlite-worker.js), checked
* with PRAGMA quick_check, and recorded in <stateDir>/backups/<service>.jsonl.
*
-- * Backups are consistent snapshots taken while the service runs; they are not a restore drill
-- * (that is Wave 22's job) and nothing here prunes old backups.
++ * Permissions: every directory under backupDir is root:root 0700 and every copy root:root 0600,
++ * with no -wal/-shm beside it. The worker writes into <backupStagingDir>/<service>-<stamp> (its own,
++ * 0700, under a root 0711 parent); root then takes that directory over, checks each copy is a plain
++ * file with one link, chowns and chmods it, and renames the directory into place.
++ *
++ * Every declared database is attempted even when an earlier one fails. A backup with any failed
++ * database is still recorded (ok: false, so a drill and retention never treat it as good) and then
++ * throws a BackupError carrying the record. A backup that found none of its databases is recorded
++ * as ok: false, empty: true and leaves no directory behind.
++ *
++ * ovhost backup --all (backupAll): every service that declares databases, one after another. A
++ * failing service never stops the others. Afterwards each service's backups are pruned to the last
++ * 7 daily and 4 weekly good ones (lib/retention.js), and a JSON summary of the run is written to
++ * <stateDir>/backup-runs/<run>.json. With --offsite the run is then uploaded (lib/offsite.js).
*/
const path = require('path');
const { service } = require('../inventory');
++const lock = require('../lock');
++const retention = require('../retention');
++
++class BackupError extends Error {
++ constructor(message, record) { super(message); this.record = record; }
++}
function stamp(ms) {
return new Date(ms).toISOString().replace(/[-:]/g, '').replace('T', '-').replace(/\.\d+Z$/, '');
}
++/** chown root:root + chmod, through the executor (root, or sudo -n). */
++async function lockDown(exec, p, mode) {
++ for (const [cmd, args] of [['chown', ['root:root', '--', p]], ['chmod', [mode.toString(8).padStart(4, '0'), '--', p]]]) {
++ const r = await exec.run(cmd, args, { privileged: true });
++ if (r.code !== 0) throw new Error(`${cmd} ${p} failed: ${r.stderr.trim()}`);
++ }
++}
++
++/** A directory that is root:root 0700, whatever it was before (older backups were 0750, service-owned). */
++async function privateDir(exec, p) {
++ await exec.mkdir(p, { owner: 'root', mode: 0o700 });
++ await lockDown(exec, p, 0o700);
++}
++
async function backup(ctx, id, { reason = 'manual' } = {}) {
const { exec, inv, log } = ctx;
const svc = service(inv, id);
if (!svc.databases.length) throw new Error(`${id} declares no databases`);
-- const dir = path.join(inv.backupDir, id, stamp(exec.now()));
-- // The worker writes as the service user, so the base must be traversable and the service's
-- // directory its own.
-- await exec.mkdir(inv.backupDir, { mode: 0o755 });
-- await exec.mkdir(path.join(inv.backupDir, id), { owner: svc.runAs, mode: 0o750 });
-- await exec.mkdir(dir, { owner: svc.runAs, mode: 0o750 });
++ const st = stamp(exec.now());
++ const dir = path.join(inv.backupDir, id, st);
++ if (await exec.stat(dir)) throw new Error(`${dir} already exists; a backup never overwrites another`);
++ // Backups hold user data: every directory under backupDir is root:root 0700 and every copy is
++ // root:root 0600. The worker must run as the service user (root never opens a service database),
++ // so it writes into a staging directory of its own, which root then takes over before the move.
++ await privateDir(exec, inv.backupDir);
++ await privateDir(exec, path.join(inv.backupDir, id));
++ await exec.mkdir(inv.backupStagingDir, { owner: 'root', mode: 0o711 });
++ await lockDown(exec, inv.backupStagingDir, 0o711);
++ const stage = path.join(inv.backupStagingDir, `${id}-${st}`);
++ if (await exec.stat(stage)) await exec.run('rm', ['-rf', '--', stage], { privileged: true });
++ await exec.mkdir(stage, { owner: svc.runAs, mode: 0o700 });
const files = [];
-- for (const db of svc.databases) {
-- if (!(await exec.stat(db.path))) { files.push({ name: db.name, source: db.path, skipped: 'not found' }); log(`skip ${db.path}: not found`); continue; }
-- // Named by the inventory name: two databases may share a file name (community.db twice).
-- const dest = path.join(dir, `${db.name}${path.extname(db.path) || '.db'}`);
-- log(`backing up ${db.path} → ${dest} (as ${svc.runAs})`);
-- const t0 = exec.now();
-- await exec.sqliteBackup(db.path, dest, { as: svc.runAs });
-- const st = await exec.stat(dest);
-- files.push({ name: db.name, source: db.path, dest, bytes: st ? st.size : null, seconds: Math.round((exec.now() - t0) / 1000) });
++ try {
++ for (const db of svc.databases) {
++ if (!(await exec.stat(db.path))) { files.push({ name: db.name, source: db.path, skipped: 'not found' }); log(`skip ${db.path}: not found`); continue; }
++ // Named by the inventory name: two databases may share a file name (community.db twice).
++ const file = `${db.name}${path.extname(db.path) || '.db'}`;
++ log(`backing up ${db.path} → ${path.join(dir, file)} (as ${svc.runAs})`);
++ const t0 = exec.now();
++ try {
++ // The worker runs PRAGMA quick_check on the copy and fails unless it answers ok.
++ await exec.sqliteBackup(db.path, path.join(stage, file), { as: svc.runAs });
++ } catch (err) {
++ files.push({ name: db.name, source: db.path, dest: path.join(dir, file), error: err.message });
++ log(`FAILED ${db.path}: ${err.message}`);
++ continue;
++ }
++ files.push({ name: db.name, source: db.path, dest: path.join(dir, file), file, check: 'quick_check ok', seconds: Math.round((exec.now() - t0) / 1000) });
++ }
++ // Take the staging directory back first: after this the service user can no longer reach
++ // anything inside it, so what is checked below cannot change underneath.
++ await lockDown(exec, stage, 0o700);
++ for (const f of files.filter((x) => x.file && !x.error)) {
++ const p = path.join(stage, f.file);
++ const fst = await exec.stat(p);
++ // A link would let the move below hand root's ownership to some other file.
++ if (!fst || !fst.isFile || fst.isSymlink || fst.nlink !== 1) { f.error = 'the copy is not a plain file'; continue; }
++ await lockDown(exec, p, 0o600);
++ f.bytes = fst.size;
++ }
++ // Anything else (a stray -wal/-shm) is not part of the backup.
++ const wanted = new Set(files.filter((x) => x.file && !x.error).map((x) => x.file));
++ for (const e of (await exec.readdir(stage)) || []) {
++ if (!wanted.has(e.name)) await exec.run('rm', ['-rf', '--', path.join(stage, e.name)], { privileged: true });
++ }
++ if (wanted.size) {
++ const mv = await exec.run('mv', ['-T', '--', stage, dir], { privileged: true });
++ if (mv.code !== 0) throw new Error(`moving ${stage} to ${dir} failed: ${mv.stderr.trim()}`);
++ await lockDown(exec, dir, 0o700);
++ }
++ } finally {
++ if (await exec.stat(stage)) await exec.run('rm', ['-rf', '--', stage], { privileged: true });
}
-- const record = { service: id, at: new Date(exec.now()).toISOString(), reason, dir, files };
++ for (const f of files) delete f.file;
++ const failed = files.filter((f) => f.error);
++ const copied = files.filter((f) => f.dest && !f.error);
++ const record = { service: id, at: new Date(exec.now()).toISOString(), reason, dir, ok: !failed.length && copied.length > 0, files };
++ // Nothing existed to back up (a service not deployed yet): no directory is left, so an empty
++ // backup can never stand in for a real one in retention or a drill.
++ if (!copied.length && !failed.length) record.empty = true;
await exec.appendFile(path.join(inv.stateDir, 'backups', `${id}.jsonl`), `${JSON.stringify(record)}\n`);
++ if (failed.length) throw new BackupError(failed.map((f) => `${f.name}: ${f.error}`).join('; '), record);
return record;
}
--module.exports = { backup, stamp };
++async function records(exec, inv, id) {
++ const text = await exec.readFile(path.join(inv.stateDir, 'backups', `${id}.jsonl`));
++ return String(text || '').split('\n').filter(Boolean).map((l) => { try { return JSON.parse(l); } catch { return null; } }).filter((r) => r && r.dir);
++}
++
++/**
++ * Prune <backupDir>/<id>/<stamp> directories. Only directories named by a stamp and recorded in the
++ * service's backup log are candidates; anything else in the directory is left alone. A directory is
++ * good when any record for it has ok !== false (records before ok existed count as good).
++ * good kept by the retention policy, else removed
++ * not good (a failed backup) removed once a newer good backup exists
++ * Nothing is removed while the service has no good backup at all.
++ */
++async function prune(ctx, id, { daily = 7, weekly = 4 } = {}) {
++ const { exec, inv, log } = ctx;
++ const base = path.join(inv.backupDir, id);
++ const entries = (await exec.readdir(base)) || [];
++ const recs = await records(exec, inv, id);
++ const recorded = new Map();
++ for (const r of recs) {
++ if (path.dirname(path.normalize(r.dir)) !== base) continue;
++ const name = path.basename(r.dir);
++ recorded.set(name, (recorded.get(name) || false) || r.ok !== false);
++ }
++ const dirs = entries.filter((e) => e.isDir && retention.STAMP_RE.test(e.name)).map((e) => e.name);
++ const good = dirs.filter((d) => recorded.get(d) === true);
++ const unrecorded = dirs.filter((d) => !recorded.has(d));
++ if (!good.length) return { kept: dirs, removed: [], unrecorded };
++ const keepSet = retention.keep(good, { daily, weekly });
++ const newestGood = good.slice().sort().pop();
++ const remove = dirs.filter((d) => recorded.has(d) && (recorded.get(d) ? !keepSet.has(d) : d < newestGood));
++ for (const d of remove) {
++ const full = path.join(base, d);
++ // Belt and braces: exactly <backupDir>/<id>/<stamp>, nothing else.
++ if (path.dirname(full) !== base || !retention.STAMP_RE.test(path.basename(full))) continue;
++ const r = await exec.run('rm', ['-rf', '--', full], { privileged: true });
++ if (r.code !== 0) throw new Error(`could not remove ${full}: ${r.stderr.trim()}`);
++ log(`pruned ${full}`);
++ }
++ return { kept: dirs.filter((d) => !remove.includes(d)), removed: remove, unrecorded };
++}
++
++async function backupAll(ctx, { daily = 7, weekly = 4, prune: doPrune = true, offsite = null } = {}) {
++ const { exec, inv, log } = ctx;
++ const held = await lock.acquire(exec, inv, '_backup-all');
++ const started = exec.now();
++ const run = stamp(started);
++ const summary = { run, host: inv.host || null, startedAt: new Date(started).toISOString(), finishedAt: null, ok: true, retention: { daily, weekly }, services: [], offsite: null };
++ const file = path.join(inv.stateDir, 'backup-runs', `${run}.json`);
++ try {
++ for (const svc of Object.values(inv.services)) {
++ if (!svc.databases.length) continue;
++ const entry = { service: svc.id, status: 'ok', dir: null, files: [], error: null, pruned: [] };
++ summary.services.push(entry);
++ try {
++ const rec = await backup(ctx, svc.id, { reason: `scheduled ${run}` });
++ entry.dir = rec.empty ? null : rec.dir;
++ entry.files = rec.files;
++ if (rec.empty) entry.status = 'skipped';
++ } catch (err) {
++ entry.status = 'failed';
++ entry.error = err.message;
++ if (err.record) { entry.dir = err.record.dir; entry.files = err.record.files; }
++ log(`✗ ${svc.id}: ${err.message}`);
++ }
++ if (doPrune) {
++ try { entry.pruned = (await prune(ctx, svc.id, { daily, weekly })).removed; } catch (err) {
++ entry.status = 'failed';
++ entry.error = [entry.error, `prune: ${err.message}`].filter(Boolean).join('; ');
++ }
++ }
++ }
++ summary.ok = !summary.services.some((s) => s.status === 'failed');
++ if (offsite) {
++ try {
++ summary.offsite = await offsite(summary);
++ } catch (err) {
++ summary.offsite = { ok: false, error: err.message };
++ }
++ if (!summary.offsite.ok) summary.ok = false;
++ }
++ } finally {
++ summary.finishedAt = new Date(exec.now()).toISOString();
++ await exec.mkdir(path.join(inv.stateDir, 'backup-runs'), { mode: 0o750 });
++ await exec.writeFile(file, `${JSON.stringify(summary, null, 2)}\n`, { mode: 0o640 });
++ await held.release();
++ }
++ summary.file = file;
++ return summary;
++}
++
++/** The summary of a backup --all run (the latest when run is null). */
++async function readRun(exec, inv, run = null) {
++ const dir = path.join(inv.stateDir, 'backup-runs');
++ let name = run;
++ if (!name) {
++ const list = ((await exec.readdir(dir)) || []).map((e) => e.name).filter((n) => /^\d{8}-\d{6}\.json$/.test(n)).sort();
++ if (!list.length) return null;
++ name = list[list.length - 1].replace(/\.json$/, '');
++ }
++ if (!retention.STAMP_RE.test(name)) throw new Error(`--run must be a run id such as 20260923-033000 (got ${JSON.stringify(name)})`);
++ const text = await exec.readFile(path.join(dir, `${name}.json`));
++ if (text == null) return null;
++ const summary = JSON.parse(text);
++ summary.file = path.join(dir, `${name}.json`);
++ return summary;
++}
++
++async function writeRun(exec, summary) {
++ const { file, ...rest } = summary;
++ await exec.writeFile(file, `${JSON.stringify(rest, null, 2)}\n`, { mode: 0o640 });
++}
++
++module.exports = { backup, backupAll, prune, records, readRun, writeRun, stamp, lockDown, BackupError };
ddiff --git a/lib/commands/drill.js b/lib/commands/drill.js
iindex 5c91629..493f9c6 100644
---- a/lib/commands/drill.js
++++ b/lib/commands/drill.js
@@@ -143,8 +143,9 @@ async function resolveBackup(exec, inv, svc, dirArg) {
dir = path.normalize(dir);
} else {
const text = await exec.readFile(path.join(inv.stateDir, 'backups', `${svc.id}.jsonl`));
-- const records = String(text || '').split('\n').filter(Boolean).map((l) => { try { return JSON.parse(l); } catch { return null; } }).filter((r) => r && r.dir);
-- if (!records.length) throw new DrillError(`no ovhost backup recorded for ${svc.id}; run \`ovhost backup ${svc.id}\` first or pass --backup <dir>`);
++ // A failed or empty backup (ok: false) is never restored.
++ const records = String(text || '').split('\n').filter(Boolean).map((l) => { try { return JSON.parse(l); } catch { return null; } }).filter((r) => r && r.dir && r.ok !== false);
++ if (!records.length) throw new DrillError(`no good ovhost backup recorded for ${svc.id}; run \`ovhost backup ${svc.id}\` first or pass --backup <dir>`);
const last = records[records.length - 1];
dir = last.dir;
takenAt = last.at || null;
@@@ -253,8 +254,10 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
// 1. restore and check every database copy
for (const f of bk.files) {
const dest = vars.db[f.name];
-- log(`restoring ${f.file} → ${dest} (as ${svc.runAs})`);
-- const cp = await exec.run('cp', ['--no-preserve=mode,ownership', '--', f.file, dest], { as: svc.runAs });
++ log(`restoring ${f.file} → ${dest} (owned by ${svc.runAs})`);
++ // Backups are root-only (0600), so root copies; install unlinks whatever is at dest
++ // before writing, so nothing planted in the drill directory is followed.
++ const cp = await exec.run('install', ['-o', svc.runAs, '-m', '0600', '-T', '--', f.file, dest], { privileged: true });
if (cp.code !== 0) throw fail('restore', `copying ${f.file} failed: ${(cp.stderr || '').trim().slice(0, 200)}`);
const st = await exec.stat(dest);
const entry = { name: f.name, source: f.file, copy: dest, bytes: st ? st.size : null, integrity: null };
ddiff --git a/lib/executor.js b/lib/executor.js
iindex 56a6986..7425cf1 100644
---- a/lib/executor.js
++++ b/lib/executor.js
@@@ -13,9 +13,11 @@
* readFile(path, { privileged }) -> string | null
* writeFile(path, content, { privileged, mode }) ; appendFile(path, text) ; removeFile(path, { privileged })
* symlink(target, path, { privileged }) ; readlink(path) -> string | null
-- * stat(path) -> { mode, uid, gid, owner, isFile, isDir, isSymlink, size } | null
++ * stat(path) -> { mode, uid, gid, owner, isFile, isDir, isSymlink, size, nlink } | null
* readdir(path) -> [{ name, isDir }] | null
* mkdir(path, { owner, mode }) ; createExclusive(path, content) -> bool ; pidAlive(pid) -> bool
++ * readStream(path) -> Readable ; writeStream(path, { mode }) -> Writable (creates the file
++ * exclusively: it fails if the path exists). Off-host backups only, which run as root.
* http(url, { timeoutMs, headers }) -> { status, body } | { status: 0, error }
* sqlite(db, sql, { as }) -> rows ; sqliteBackup(db, dest, { as })
* listeners(port) -> [{ pid, process }]
@@@ -133,7 +135,7 @@ function createSystemExecutor({ env = process.env } = {}) {
try {
const l = fs.lstatSync(p);
const s = l.isSymbolicLink() ? fs.statSync(p) : l;
-- return { mode: s.mode & 0o7777, uid: s.uid, gid: s.gid, owner: userById(s.uid), isFile: s.isFile(), isDir: s.isDirectory(), isSymlink: l.isSymbolicLink(), size: s.size };
++ return { mode: s.mode & 0o7777, uid: s.uid, gid: s.gid, owner: userById(s.uid), isFile: s.isFile(), isDir: s.isDirectory(), isSymlink: l.isSymbolicLink(), size: s.size, nlink: l.nlink };
} catch { return null; }
}
@@@ -209,6 +211,8 @@ function createSystemExecutor({ env = process.env } = {}) {
return {
kind: 'system',
++ readStream: (p) => fs.createReadStream(p),
++ writeStream: (p, { mode = 0o600 } = {}) => fs.createWriteStream(p, { flags: 'wx', mode }),
run,
readFile,
writeFile,
ddiff --git a/lib/index.js b/lib/index.js
iindex c7d2ff4..601eb36 100644
---- a/lib/index.js
++++ b/lib/index.js
@@@ -7,7 +7,8 @@ const inventory = require('./inventory');
const ops = require('./release-ops');
const { validate, envNames } = require('./commands/validate');
const { status } = require('./commands/status');
--const { backup } = require('./commands/backup');
++const { backup, backupAll, readRun, writeRun, BackupError } = require('./commands/backup');
++const offsite = require('./offsite');
const { snapshot } = require('./commands/snapshot');
const { drill, DrillError } = require('./commands/drill');
const releases = require('./releases');
@@@ -25,6 +26,11 @@ module.exports = {
rollback: ops.rollback,
status,
backup,
++ backupAll,
++ readRun,
++ writeRun,
++ BackupError,
++ offsite,
snapshot,
drill,
DrillError,
ddiff --git a/lib/inventory.js b/lib/inventory.js
iindex f45f89c..4b61ea4 100644
---- a/lib/inventory.js
++++ b/lib/inventory.js
@@@ -211,6 +211,9 @@ function normalise(raw) {
host: raw.host || null,
stateDir: abs(raw.stateDir || '/var/lib/openvibe-host', 'stateDir'),
backupDir: abs(raw.backupDir || '/var/backups/openvibe', 'backupDir'),
++ // Where the service user's backup worker writes before root takes the copy over (see
++ // commands/backup.js). The same filesystem as backupDir, so the final move is a rename.
++ backupStagingDir: abs(raw.backupStagingDir || `${path.normalize(raw.backupDir || '/var/backups/openvibe').replace(/\/+$/, '')}.staging`, 'backupStagingDir'),
drillDir: abs(raw.drillDir || '/var/lib/openvibe-drills', 'drillDir'),
nginx: {
sitesAvailable: abs((raw.nginx && raw.nginx.sitesAvailable) || '/etc/nginx/sites-available', 'nginx.sitesAvailable'),
ddiff --git a/lib/offsite.js b/lib/offsite.js
nnew file mode 100644
iindex 0000000..190206e
---- /dev/null
++++ b/lib/offsite.js
@@@ -0,0 +1,424 @@
++'use strict';
++/**
++ * Off-host copies of `ovhost backup --all` runs: every backed-up database is encrypted on the host
++ * (lib/backup-crypto.js, AES-256-GCM with a key that stays on the host) and uploaded to S3-compatible
++ * object storage (Backblaze B2 in production, the same API Media uses). Nothing is uploaded in the
++ * clear except the run manifest, which holds names, sizes and hashes, never data or secrets, and
++ * carries an HMAC made with the backup key.
++ *
++ * Configuration comes from a root-only env file (default /etc/openvibe/backup.env):
++ *
++ * BACKUP_S3_ENDPOINT https://s3.<region>.backblazeb2.com (any S3-compatible endpoint)
++ * BACKUP_S3_BUCKET bucket name
++ * BACKUP_S3_PREFIX key prefix, default openvibe-backups
++ * BACKUP_S3_KEY_ID access key id (B2: application key id)
++ * BACKUP_S3_SECRET secret access key (B2: application key)
++ * BACKUP_ENCRYPTION_KEY_FILE root-only file with the 32-byte key as 64 hex characters
++ * BACKUP_S3_REGION optional; derived from a B2 endpoint, else us-east-1
++ * BACKUP_S3_FORCE_PATH_STYLE optional, default 1 (B2 needs path-style)
++ * BACKUP_OFFSITE_RETENTION_DAYS optional, default 30
++ * BACKUP_OFFSITE_PRUNE optional, default 1; 0 when a bucket lifecycle rule does it
++ *
++ * This file is the only place in ovhost that reads env VALUES, and only these names. Values are
++ * never logged, printed, stored or put in an error message.
++ *
++ * Object layout: <prefix>/<host>/<run>/<service>/<file>.ovbk and <prefix>/<host>/<run>/manifest.json.
++ */
++const path = require('path');
++const { pipeline } = require('stream/promises');
++const bcrypto = require('./backup-crypto');
++const { STAMP_RE, parseStamp } = require('./retention');
++
++const DEFAULT_ENV_FILE = '/etc/openvibe/backup.env';
++const DEFAULT_PREFIX = 'openvibe-backups';
++const DEFAULT_RESTORE_DIR = '/var/lib/openvibe-restore';
++const PART_SIZE = 64 * 1024 * 1024;
++const MIN_KEEP_RUNS = 7;
++const PREFIX_RE = /^[A-Za-z0-9._-]+(\/[A-Za-z0-9._-]+)*$/;
++const SEGMENT_RE = /^[A-Za-z0-9._-]+$/;
++const LINE_RE = /^\s*(?:export\s+)?([A-Za-z_][A-Za-z0-9_]*)\s*=(.*)$/;
++
++class OffsiteError extends Error {
++ constructor(message, exitCode = 2) { super(message); this.exitCode = exitCode; }
++}
++
++/** KEY=value lines → { BACKUP_*: value }. Quotes are stripped; other names are ignored. */
++function parseValues(text) {
++ const out = {};
++ for (const line of String(text || '').split('\n')) {
++ const m = line.match(LINE_RE);
++ if (!m || !m[1].startsWith('BACKUP_')) continue;
++ let v = m[2].trim();
++ if ((v.startsWith('"') && v.endsWith('"') && v.length >= 2) || (v.startsWith("'") && v.endsWith("'") && v.length >= 2)) v = v.slice(1, -1);
++ else v = v.replace(/\s+#.*$/, '');
++ out[m[1]] = v;
++ }
++ return out;
++}
++
++/** A root-owned file nobody else can read or write (when ovhost runs as root). */
++async function checkPrivateFile(exec, file, what) {
++ const st = await exec.stat(file);
++ if (!st || !st.isFile) throw new OffsiteError(`${what} ${file} not found`, 1);
++ if (await exec.isRoot()) {
++ if (st.uid !== 0) throw new OffsiteError(`${what} ${file} must be owned by root`, 1);
++ if (st.mode & 0o077) throw new OffsiteError(`${what} ${file} must not be readable or writable by group or others (chmod 0600)`, 1);
++ }
++}
++
++async function loadConfig(exec, { file = null, env = {} } = {}) {
++ if (!(await exec.isRoot())) throw new OffsiteError('off-host backups run as root (the backup env file and the key are root-only): use sudo', 1);
++ const f = file || env.OVHOST_BACKUP_ENV || DEFAULT_ENV_FILE;
++ if (!path.isAbsolute(f)) throw new OffsiteError('the backup env file must be an absolute path', 1);
++ await checkPrivateFile(exec, f, 'backup env file');
++ const v = parseValues(await exec.readFile(f, { privileged: true }));
++ const missing = ['BACKUP_S3_ENDPOINT', 'BACKUP_S3_BUCKET', 'BACKUP_S3_KEY_ID', 'BACKUP_S3_SECRET', 'BACKUP_ENCRYPTION_KEY_FILE'].filter((n) => !v[n]);
++ if (missing.length) throw new OffsiteError(`${f} is missing or has empty: ${missing.join(', ')}`, 1);
++ let endpoint;
++ try { endpoint = new URL(v.BACKUP_S3_ENDPOINT); } catch { throw new OffsiteError('BACKUP_S3_ENDPOINT is not a URL', 1); }
++ const loopback = ['127.0.0.1', 'localhost', '[::1]'].includes(endpoint.hostname);
++ if (endpoint.protocol !== 'https:' && !(endpoint.protocol === 'http:' && loopback)) throw new OffsiteError('BACKUP_S3_ENDPOINT must be https:// (plain http only on loopback)', 1);
++ if (endpoint.username || endpoint.password) throw new OffsiteError('BACKUP_S3_ENDPOINT must not carry credentials; use BACKUP_S3_KEY_ID and BACKUP_S3_SECRET', 1);
++ const b2 = /^s3\.([a-z0-9-]+)\.backblazeb2\.com$/.exec(endpoint.hostname);
++ const prefix = (v.BACKUP_S3_PREFIX || DEFAULT_PREFIX).replace(/^\/+|\/+$/g, '');
++ if (!PREFIX_RE.test(prefix)) throw new OffsiteError('BACKUP_S3_PREFIX may only use letters, digits, . _ - and /', 1);
++ if (!/^[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]$/.test(v.BACKUP_S3_BUCKET)) throw new OffsiteError('BACKUP_S3_BUCKET is not a bucket name', 1);
++ const keyFile = v.BACKUP_ENCRYPTION_KEY_FILE;
++ if (!path.isAbsolute(keyFile)) throw new OffsiteError('BACKUP_ENCRYPTION_KEY_FILE must be an absolute path', 1);
++ const days = v.BACKUP_OFFSITE_RETENTION_DAYS ? Number(v.BACKUP_OFFSITE_RETENTION_DAYS) : 30;
++ if (!(Number.isInteger(days) && days >= 1)) throw new OffsiteError('BACKUP_OFFSITE_RETENTION_DAYS must be a whole number of days', 1);
++ const cfg = {
++ envFile: f,
++ endpoint: endpoint.origin,
++ region: v.BACKUP_S3_REGION || (b2 ? b2[1] : 'us-east-1'),
++ bucket: v.BACKUP_S3_BUCKET,
++ prefix,
++ forcePathStyle: !['0', 'false', 'no'].includes(String(v.BACKUP_S3_FORCE_PATH_STYLE || '1').toLowerCase()),
++ keyFile,
++ retentionDays: days,
++ prune: !['0', 'false', 'no'].includes(String(v.BACKUP_OFFSITE_PRUNE || '1').toLowerCase()),
++ };
++ // Credentials are not enumerable: JSON.stringify(cfg) and util.inspect never show them.
++ Object.defineProperty(cfg, 'credentials', { value: { accessKeyId: v.BACKUP_S3_KEY_ID, secretAccessKey: v.BACKUP_S3_SECRET }, enumerable: false });
++ return cfg;
++}
++
++async function loadKey(exec, cfg) {
++ await checkPrivateFile(exec, cfg.keyFile, 'encryption key file');
++ try { return bcrypto.parseKey(await exec.readFile(cfg.keyFile, { privileged: true })); } catch (err) { throw new OffsiteError(err.message, 1); }
++}
++
++/** The real S3 client (@aws-sdk/client-s3). Tests pass their own { send(command) }. */
++function createClient(cfg) {
++ const { S3Client } = require('@aws-sdk/client-s3');
++ return new S3Client({
++ region: cfg.region,
++ endpoint: cfg.endpoint,
++ forcePathStyle: cfg.forcePathStyle,
++ credentials: cfg.credentials,
++ maxAttempts: 5,
++ // Checksums only where S3 requires them: every object is already authenticated by GCM and
++ // by the SHA-256 in the manifest, and not every S3-compatible store takes the newer headers.
++ requestChecksumCalculation: 'WHEN_REQUIRED',
++ responseChecksumValidation: 'WHEN_REQUIRED',
++ });
++}
++
++function S3() { return require('@aws-sdk/client-s3'); }
++
++/** Where a run lives: <prefix>/<host>/<run>/. */
++function runPrefix(cfg, host, run) { return `${cfg.prefix}/${host}/${run}/`; }
++
++function describe(cfg) {
++ return { endpoint: cfg.endpoint, bucket: cfg.bucket, prefix: cfg.prefix, retentionDays: cfg.retentionDays, prune: cfg.prune };
++}
++
++/**
++ * Upload an async iterable of Buffers. One PutObject when it is smaller than one part, else a
++ * multipart upload in parts of exactly `partSize` (the last one shorter), aborted on failure so no
++ * orphaned parts are billed. Memory use is about one part.
++ */
++async function uploadIterable(client, bucket, key, source, { partSize = PART_SIZE, contentType = 'application/octet-stream' } = {}) {
++ const s3 = S3();
++ let pending = [];
++ let pendingLen = 0;
++ let uploadId = null;
++ const parts = [];
++ const sendPart = async (buf) => {
++ const PartNumber = parts.length + 1;
++ const r = await client.send(new s3.UploadPartCommand({ Bucket: bucket, Key: key, UploadId: uploadId, PartNumber, Body: buf, ContentLength: buf.length }));
++ parts.push({ ETag: r && r.ETag, PartNumber });
++ };
++ try {
++ for await (const chunk of source) {
++ pending.push(chunk);
++ pendingLen += chunk.length;
++ while (pendingLen >= partSize) {
++ // Every part but the last is exactly partSize.
++ const all = Buffer.concat(pending);
++ const buf = all.subarray(0, partSize);
++ pending = [all.subarray(partSize)];
++ pendingLen = all.length - partSize;
++ if (!uploadId) {
++ const r = await client.send(new s3.CreateMultipartUploadCommand({ Bucket: bucket, Key: key, ContentType: contentType }));
++ uploadId = r && r.UploadId;
++ if (!uploadId) throw new Error('CreateMultipartUpload returned no UploadId');
++ }
++ await sendPart(buf);
++ }
++ }
++ const rest = Buffer.concat(pending);
++ if (!uploadId) {
++ await client.send(new s3.PutObjectCommand({ Bucket: bucket, Key: key, Body: rest, ContentLength: rest.length, ContentType: contentType }));
++ return { parts: 1 };
++ }
++ if (rest.length) await sendPart(rest);
++ await client.send(new s3.CompleteMultipartUploadCommand({ Bucket: bucket, Key: key, UploadId: uploadId, MultipartUpload: { Parts: parts } }));
++ return { parts: parts.length };
++ } catch (err) {
++ if (uploadId) {
++ try { await client.send(new s3.AbortMultipartUploadCommand({ Bucket: bucket, Key: key, UploadId: uploadId })); } catch { /* the original error matters more */ }
++ }
++ throw err;
++ }
++}
++
++async function listAll(client, bucket, prefix) {
++ const s3 = S3();
++ const out = [];
++ let token;
++ do {
++ const r = await client.send(new s3.ListObjectsV2Command({ Bucket: bucket, Prefix: prefix, ContinuationToken: token }));
++ for (const o of (r && r.Contents) || []) out.push({ key: o.Key, size: o.Size || 0, lastModified: o.LastModified || null });
++ token = r && r.IsTruncated ? r.NextContinuationToken : undefined;
++ } while (token);
++ return out;
++}
++
++/** Runs stored off-host for a host: [{ run, objects, bytes, manifest, services }], newest first. */
++async function listRuns(client, cfg, host) {
++ const base = `${cfg.prefix}/${host}/`;
++ const runs = new Map();
++ for (const o of await listAll(client, cfg.bucket, base)) {
++ const rest = o.key.slice(base.length).split('/');
++ if (!STAMP_RE.test(rest[0])) continue;
++ const r = runs.get(rest[0]) || { run: rest[0], objects: 0, bytes: 0, manifest: false, services: new Set(), keys: [] };
++ r.objects += 1;
++ r.bytes += o.size;
++ r.keys.push(o.key);
++ if (rest.length === 2 && rest[1] === 'manifest.json') r.manifest = true;
++ if (rest.length === 3) r.services.add(rest[1]);
++ runs.set(rest[0], r);
++ }
++ return [...runs.values()].sort((a, b) => (a.run < b.run ? 1 : -1)).map((r) => ({ ...r, services: [...r.services].sort() }));
++}
++
++async function hostName(exec, inv, override) {
++ const h = override || inv.host || await exec.hostname();
++ if (!SEGMENT_RE.test(h)) throw new OffsiteError(`host name ${JSON.stringify(h)} cannot be used in an object key`, 1);
++ return h;
++}
++
++function manifestBody(m) {
++ const { hmac, ...rest } = m;
++ return JSON.stringify(rest);
++}
++
++/**
++ * Encrypt and upload every database copy a backup run made. Returns the `offsite` section of the
++ * run summary. Remote runs older than the retention are pruned only after a run that uploaded
++ * everything, and the newest MIN_KEEP_RUNS runs are always kept, so failing uploads never let the
++ * last good copies age out.
++ */
++async function push(ctx, summary, { cfg, client, partSize = PART_SIZE, chunkSize } = {}) {
++ const { exec, inv, log } = ctx;
++ const master = await loadKey(exec, cfg);
++ const host = await hostName(exec, inv, summary.host);
++ const base = runPrefix(cfg, host, summary.run);
++ const result = { ok: true, ...describe(cfg), run: summary.run, host, location: base, keyId: bcrypto.keyId(master), uploaded: [], failures: [], pruned: [], manifest: null };
++ const manifest = { format: 'openvibe-backup-manifest/1', encryption: 'OVBKAES1', host, run: summary.run, createdAt: new Date(exec.now()).toISOString(), keyId: result.keyId, services: {} };
++ for (const svc of summary.services || []) {
++ for (const f of svc.files || []) {
++ if (!f.dest || f.error) continue;
++ const name = path.basename(f.dest);
++ if (!SEGMENT_RE.test(svc.service) || !SEGMENT_RE.test(name)) { result.failures.push({ service: svc.service, file: name, error: 'unsafe name' }); continue; }
++ const key = `${base}${svc.service}/${name}.ovbk`;
++ const plain = bcrypto.createDigestTap();
++ const cipher = bcrypto.createDigestTap();
++ const t0 = exec.now();
++ try {
++ let up = null;
++ await pipeline(exec.readStream(f.dest), plain, bcrypto.createEncryptStream(master, chunkSize ? { chunkSize } : {}), cipher, async (source) => { up = await uploadIterable(client, cfg.bucket, key, source, { partSize }); });
++ const p = plain.result();
++ const c = cipher.result();
++ const entry = { service: svc.service, name: f.name, file: name, object: key, source: f.source || null, plainBytes: p.bytes, plainSha256: p.sha256, cipherBytes: c.bytes, cipherSha256: c.sha256, parts: up.parts, seconds: Math.round((exec.now() - t0) / 1000) };
++ result.uploaded.push(entry);
++ (manifest.services[svc.service] = manifest.services[svc.service] || { files: [] }).files.push(entry);
++ log(`uploaded ${f.dest} → s3://${cfg.bucket}/${key} (${c.bytes} bytes encrypted)`);
++ } catch (err) {
++ result.failures.push({ service: svc.service, file: name, error: err.message });
++ log(`✗ upload ${f.dest}: ${err.message}`);
++ }
++ }
++ }
++ manifest.failures = result.failures;
++ manifest.hmac = bcrypto.hmacManifest(master, manifestBody(manifest));
++ const manifestKey = `${base}manifest.json`;
++ try {
++ const body = Buffer.from(`${JSON.stringify(manifest, null, 2)}\n`);
++ await uploadIterable(client, cfg.bucket, manifestKey, [body], { contentType: 'application/json' });
++ result.manifest = manifestKey;
++ } catch (err) {
++ result.failures.push({ file: 'manifest.json', error: err.message });
++ }
++ result.ok = result.failures.length === 0;
++ if (result.ok && cfg.prune) {
++ try { result.pruned = await pruneRemote(ctx, { cfg, client, host, current: summary.run }); } catch (err) {
++ result.ok = false;
++ result.failures.push({ file: '(prune)', error: err.message });
++ }
++ } else if (!result.ok) log('off-host prune skipped: this run did not upload everything');
++ return result;
++}
++
++async function pruneRemote(ctx, { cfg, client, host, current }) {
++ const { exec, log } = ctx;
++ const cutoff = exec.now() - cfg.retentionDays * 86400000;
++ const runs = await listRuns(client, cfg, host);
++ const protectedRuns = new Set(runs.filter((r) => r.manifest).slice(0, MIN_KEEP_RUNS).map((r) => r.run));
++ protectedRuns.add(current);
++ const s3 = S3();
++ const pruned = [];
++ for (const r of runs) {
++ if (protectedRuns.has(r.run)) continue;
++ const at = parseStamp(r.run);
++ if (at === null || at >= cutoff) continue;
++ for (const key of r.keys) {
++ if (!key.startsWith(runPrefix(cfg, host, r.run))) continue;
++ await client.send(new s3.DeleteObjectCommand({ Bucket: cfg.bucket, Key: key }));
++ }
++ pruned.push(r.run);
++ log(`pruned off-host run ${r.run} (${r.objects} objects)`);
++ }
++ return pruned;
++}
++
++async function getObjectStream(client, bucket, key) {
++ const r = await client.send(new (S3().GetObjectCommand)({ Bucket: bucket, Key: key }));
++ if (!r || !r.Body) throw new Error(`empty response for ${key}`);
++ return r.Body;
++}
++
++async function readManifest(client, cfg, master, host, run) {
++ const key = `${runPrefix(cfg, host, run)}manifest.json`;
++ let text = '';
++ const body = await getObjectStream(client, cfg.bucket, key);
++ for await (const c of body) text += Buffer.isBuffer(c) ? c.toString('utf8') : String(c);
++ let m;
++ try { m = JSON.parse(text); } catch { throw new OffsiteError(`${key} is not JSON`); }
++ if (!m || m.hmac !== bcrypto.hmacManifest(master, manifestBody(m))) throw new OffsiteError(`${key} failed its HMAC check: it was altered, or it was made with another key (this host's key id ${bcrypto.keyId(master)}, manifest says ${m && m.keyId})`);
++ if (m.run !== run || m.host !== host) throw new OffsiteError(`${key} describes ${m.host}/${m.run}, not ${host}/${run}`);
++ return m;
++}
++
++/** Directories restore-download must never write into: anything holding a declared database. */
++function unsafeOut(inv, out) {
++ for (const svc of Object.values(inv.services)) {
++ for (const db of svc.databases) {
++ const dir = path.dirname(db.path);
++ if (out === dir || db.path.startsWith(`${out}/`) || out.startsWith(`${dir}/`)) return `${out} holds or sits inside ${svc.id}'s database directory ${dir}`;
++ }
++ if (out === svc.repo || out.startsWith(`${svc.repo}/`) || svc.repo.startsWith(`${out}/`)) return `${out} overlaps ${svc.id}'s checkout ${svc.repo}`;
++ }
++ for (const d of [inv.backupDir, inv.stateDir]) if (out === d || d.startsWith(`${out}/`) || out.startsWith(`${d}/`)) return `${out} overlaps ${d}`;
++ return null;
++}
++
++/**
++ * Download one service's databases from an off-host run, verify and decrypt them into a new
++ * directory (root:root 0700, files 0600), and quick_check each. It never touches a live database: the directory
++ * must not exist yet (or be empty), must not overlap any service's checkout or database directory,
++ * and every file is created exclusively. Putting a copy in place is a separate, manual step.
++ */
++async function restoreDownload(ctx, id, runArg, { cfg, client, out = null, host: hostArg = null } = {}) {
++ const { exec, inv, log } = ctx;
++ if (!SEGMENT_RE.test(id)) throw new OffsiteError(`bad service name ${JSON.stringify(id)}`, 1);
++ const master = await loadKey(exec, cfg);
++ const host = await hostName(exec, inv, hostArg);
++ let run = runArg;
++ if (!run || run === 'latest') {
++ const runs = (await listRuns(client, cfg, host)).filter((r) => r.manifest && r.services.includes(id));
++ if (!runs.length) throw new OffsiteError(`no off-host run of ${id} under s3://${cfg.bucket}/${cfg.prefix}/${host}/`, 1);
++ run = runs[0].run;
++ }
++ if (!STAMP_RE.test(run)) throw new OffsiteError(`<run> must be a run id such as 20260923-033000, or latest (got ${JSON.stringify(run)})`, 1);
++ const manifest = await readManifest(client, cfg, master, host, run);
++ const files = ((manifest.services[id] || {}).files) || [];
++ if (!files.length) throw new OffsiteError(`run ${run} has no files for ${id} (it has: ${Object.keys(manifest.services).join(', ') || 'none'})`, 1);
++ const dir = path.normalize(out || path.join(DEFAULT_RESTORE_DIR, `${id}-${run}`));
++ if (!path.isAbsolute(dir)) throw new OffsiteError('--out must be an absolute path', 1);
++ const why = unsafeOut(inv, dir);
++ if (why) throw new OffsiteError(`refusing --out: ${why}`, 1);
++ const existing = await exec.readdir(dir);
++ if (existing && existing.length) throw new OffsiteError(`${dir} already exists and is not empty; restore-download never overwrites`, 1);
++ if (!existing && await exec.stat(dir)) throw new OffsiteError(`${dir} exists and is not a directory`, 1);
++ const { lockDown } = require('./commands/backup');
++ if (!out) { await exec.mkdir(DEFAULT_RESTORE_DIR, { owner: 'root', mode: 0o700 }); await lockDown(exec, DEFAULT_RESTORE_DIR, 0o700); }
++ await exec.mkdir(dir, { owner: 'root', mode: 0o700 });
++ await lockDown(exec, dir, 0o700);
++ const record = { service: id, host, run, dir, keyId: manifest.keyId, files: [] };
++ for (const f of files) {
++ if (!SEGMENT_RE.test(f.file || '') || f.object !== `${runPrefix(cfg, host, run)}${id}/${f.file}.ovbk`) throw new OffsiteError(`manifest entry ${JSON.stringify(f.file)} has an unexpected name or object key`);
++ const dest = path.join(dir, f.file);
++ const cipher = bcrypto.createDigestTap();
++ const plain = bcrypto.createDigestTap();
++ log(`downloading s3://${cfg.bucket}/${f.object} → ${dest}`);
++ try {
++ await pipeline(await getObjectStream(client, cfg.bucket, f.object), cipher, bcrypto.createDecryptStream(master), plain, exec.writeStream(dest, { mode: 0o600 }));
++ } catch (err) {
++ await exec.removeFile(dest);
++ throw new OffsiteError(`${f.file}: ${err.message}`);
++ }
++ const c = cipher.result();
++ const p = plain.result();
++ if (c.sha256 !== f.cipherSha256 || p.sha256 !== f.plainSha256 || p.bytes !== f.plainBytes) {
++ await exec.removeFile(dest);
++ throw new OffsiteError(`${f.file}: the download does not match the manifest's hashes`);
++ }
++ let check;
++ try {
++ const rows = await exec.sqlite(dest, 'PRAGMA quick_check');
++ check = rows && rows[0] ? String(Object.values(rows[0])[0]) : 'no answer';
++ } catch (err) { check = `error: ${err.message}`; }
++ record.files.push({ name: f.name, file: dest, source: f.source, bytes: p.bytes, sha256: p.sha256, quickCheck: check });
++ if (check !== 'ok') throw new OffsiteError(`${dest}: quick_check = ${check}`);
++ }
++ return record;
++}
++
++/** Configuration, key and bucket access, without uploading anything. */
++async function check(ctx, { cfg, client }) {
++ const { exec, inv } = ctx;
++ const master = await loadKey(exec, cfg);
++ const host = await hostName(exec, inv);
++ const s3 = S3();
++ await client.send(new s3.ListObjectsV2Command({ Bucket: cfg.bucket, Prefix: `${cfg.prefix}/${host}/`, MaxKeys: 1 }));
++ return { ok: true, ...describe(cfg), region: cfg.region, host, keyId: bcrypto.keyId(master) };
++}
++
++module.exports = {
++ loadConfig,
++ createClient,
++ push,
++ pruneRemote,
++ listRuns,
++ restoreDownload,
++ check,
++ uploadIterable,
++ parseValues,
++ hostName,
++ OffsiteError,
++ DEFAULT_ENV_FILE,
++ PART_SIZE,
++ MIN_KEEP_RUNS,
++};
ddiff --git a/lib/retention.js b/lib/retention.js
nnew file mode 100644
iindex 0000000..f7f35b9
---- /dev/null
++++ b/lib/retention.js
@@@ -0,0 +1,44 @@
++'use strict';
++/**
++ * Backup retention: which dated backups to keep. Pure functions over stamps (YYYYMMDD-HHMMSS, UTC),
++ * so the policy is tested on its own.
++ *
++ * keep(): the newest backup of each of the last `daily` distinct days that have a backup, and the
++ * newest backup of each of the last `weekly` distinct ISO weeks that have one. Days and weeks
++ * without a backup do not use up a slot, so a service whose backups have been failing for a month
++ * still keeps its last good ones.
++ */
++const STAMP_RE = /^(\d{4})(\d{2})(\d{2})-(\d{2})(\d{2})(\d{2})$/;
++
++function parseStamp(s) {
++ const m = STAMP_RE.exec(String(s));
++ if (!m) return null;
++ const ms = Date.UTC(+m[1], +m[2] - 1, +m[3], +m[4], +m[5], +m[6]);
++ return Number.isFinite(ms) ? ms : null;
++}
++
++/** ISO-8601 week key, e.g. "2026-W39". */
++function isoWeek(ms) {
++ const d = new Date(ms);
++ const day = (d.getUTCDay() + 6) % 7; // Monday = 0
++ const thursday = new Date(Date.UTC(d.getUTCFullYear(), d.getUTCMonth(), d.getUTCDate() - day + 3));
++ // The ISO week is the one holding that week's Thursday, numbered by the Thursday's day of year.
++ const week = Math.floor((thursday - Date.UTC(thursday.getUTCFullYear(), 0, 1)) / 86400000 / 7) + 1;
++ return `${thursday.getUTCFullYear()}-W${String(week).padStart(2, '0')}`;
++}
++
++function keep(stamps, { daily = 7, weekly = 4 } = {}) {
++ const sorted = [...new Set(stamps)].filter((s) => parseStamp(s) !== null).sort().reverse();
++ const kept = new Set();
++ const days = new Set();
++ const weeks = new Set();
++ for (const s of sorted) {
++ const day = s.slice(0, 8);
++ const week = isoWeek(parseStamp(s));
++ if (!days.has(day) && days.size < daily) { days.add(day); kept.add(s); }
++ if (!weeks.has(week) && weeks.size < weekly) { weeks.add(week); kept.add(s); }
++ }
++ return kept;
++}
++
++module.exports = { keep, parseStamp, isoWeek, STAMP_RE };
ddiff --git a/lib/sqlite-worker.js b/lib/sqlite-worker.js
iindex ed505b9..2369428 100644
---- a/lib/sqlite-worker.js
++++ b/lib/sqlite-worker.js
@@@ -10,9 +10,15 @@
*
* The source database is always opened read-only (fileMustExist); a query that tries to write fails.
* Prints one JSON line: { ok: true, rows } | { ok: true, bytes } | { ok: false, error }.
++ *
++ * Backup copies hold user data, so the worker runs with umask 077 and leaves exactly one file,
++ * mode 0600: the copy is switched to journal_mode=DELETE (a WAL source gives a WAL-mode copy), which
++ * checkpoints and removes its -wal/-shm, and any side file still present is deleted.
*/
const fs = require('fs');
++process.umask(0o077);
++
function done(obj, code = 0) {
process.stdout.write(`${JSON.stringify(obj)}\n`);
process.exit(code);
@@@ -44,9 +50,12 @@ if (req.op === 'query') {
if (!req.dest) done({ ok: false, error: 'dest required' }, 2);
if (fs.existsSync(req.dest)) done({ ok: false, error: `refusing to overwrite ${req.dest}` }, 1);
db.backup(req.dest).then(() => {
-- const check = new Database(req.dest, { readonly: true });
++ const check = new Database(req.dest);
++ check.pragma('journal_mode = DELETE');
const result = check.pragma('quick_check', { simple: true });
check.close();
++ for (const side of ['-wal', '-shm', '-journal']) fs.rmSync(`${req.dest}${side}`, { force: true });
++ fs.chmodSync(req.dest, 0o600);
if (result !== 'ok') done({ ok: false, error: `backup failed quick_check: ${result}` }, 1);
done({ ok: true, bytes: fs.statSync(req.dest).size });
}, (err) => done({ ok: false, error: err.message }, 1));
ddiff --git a/package-lock.json b/package-lock.json
iindex b304518..2ec5ede 100644
---- a/package-lock.json
++++ b/package-lock.json
@@@ -9,6 +9,7 @@
"version": "0.2.0",
"license": "AGPL-3.0-only",
"dependencies": {
++ "@aws-sdk/client-s3": "^3.1138.0",
"better-sqlite3": "^11.10.0",
"busboy": "^1.6.0",
"cookie-parser": "^1.4.7",
@@@ -29,6 +30,395 @@
"node": ">=22.0.0"
}
},
++ "node_modules/@aws-sdk/checksums": {
++ "version": "3.1001.1",
++ "resolved": "https://registry.npmjs.org/@aws-sdk/checksums/-/checksums-3.1001.1.tgz",
++ "integrity": "sha512-x12Q17KYlJAd3nKf8LV5LV0vt8sh8/6YfQLGPtrGnQf/tW4jqxPGq5GPpuVitpQYM3eUR4XB7CbxZf751NMbLw==",
++ "license": "Apache-2.0",
++ "dependencies": {
++ "@aws-sdk/core": "^3.978.1",
++ "@aws-sdk/types": "^3.974.6",
++ "@smithy/core": "^3.35.0",
++ "@smithy/types": "^4.19.0",
++ "tslib": "^2.6.2"
++ },
++ "engines": {
++ "node": ">=20.0.0"
++ }
++ },
++ "node_modules/@aws-sdk/client-s3": {
++ "version": "3.1138.0",
++ "resolved": "https://registry.npmjs.org/@aws-sdk/client-s3/-/client-s3-3.1138.0.tgz",
++ "integrity": "sha512-U7+dA10/2mk9PCoZ4JknQhGaw0atZfEtxMHT+vRGFS8z7fO8bb+Grvq5B4XkiL5epyrWImv8KwLclYWMq/4znA==",
++ "license": "Apache-2.0",
++ "dependencies": {
++ "@aws-sdk/checksums": "^3.1001.1",
++ "@aws-sdk/core": "^3.978.1",
++ "@aws-sdk/credential-provider-node": "^3.972.84",
++ "@aws-sdk/middleware-sdk-s3": "^3.972.77",
++ "@aws-sdk/signature-v4-multi-region": "^3.996.47",
++ "@aws-sdk/types": "^3.974.6",
++ "@smithy/core": "^3.35.0",
++ "@smithy/fetch-http-handler": "^5.8.0",
++ "@smithy/node-http-handler": "^4.12.1",
++ "@smithy/types": "^4.19.0",
++ "tslib": "^2.6.2"
++ },
++ "engines": {
++ "node": ">=20.0.0"
++ }
++ },
++ "node_modules/@aws-sdk/core": {
++ "version": "3.978.1",
++ "resolved": "https://registry.npmjs.org/@aws-sdk/core/-/core-3.978.1.tgz",
++ "integrity": "sha512-LbY9aGsEiznDWmUc30Nwv3aIX/+dbwTx8KfS0yOC3NPYMO+O91e6jkT1azf34FwjOndq8/Q+RcVVZz5xnerwdg==",
++ "license": "Apache-2.0",
++ "dependencies": {
++ "@aws-sdk/types": "^3.974.6",
++ "@aws-sdk/xml-builder": "^3.972.41",
++ "@aws/lambda-invoke-store": "^0.3.0",
++ "@smithy/core": "^3.35.0",
++ "@smithy/signature-v4": "^5.7.3",
++ "@smithy/types": "^4.19.0",
++ "bowser": "^2.11.0",
++ "tslib": "^2.6.2"
++ },
++ "engines": {
++ "node": ">=20.0.0"
++ }
++ },
++ "node_modules/@aws-sdk/credential-provider-env": {
++ "version": "3.972.72",
++ "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-env/-/credential-provider-env-3.972.72.tgz",
++ "integrity": "sha512-xTKO/FWJPozTIXbozVnVGoNBhaGba8TBcx+KyUjRVeOlXE+dUc7GTR1cLvu0uTdIdmemzaFbqqCshXeZA1fZew==",
++ "license": "Apache-2.0",
++ "dependencies": {
++ "@aws-sdk/core": "^3.978.1",
++ "@aws-sdk/types": "^3.974.6",
++ "@smithy/core": "^3.35.0",
++ "@smithy/types": "^4.19.0",
++ "tslib": "^2.6.2"
++ },
++ "engines": {
++ "node": ">=20.0.0"
++ }
++ },
++ "node_modules/@aws-sdk/credential-provider-http": {
++ "version": "3.972.74",
++ "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-http/-/credential-provider-http-3.972.74.tgz",
++ "integrity": "sha512-u91E/hT8f4d1xy0Jl7VG4nVKJ3lxbrZkoBTeSVoJdWBiSEUMwMS/9+e0H/aJVQV//Lt5wuzP+E69v4aRSsNTmw==",
++ "license": "Apache-2.0",
++ "dependencies": {
++ "@aws-sdk/core": "^3.978.1",
++ "@aws-sdk/types": "^3.974.6",
++ "@smithy/core": "^3.35.0",
++ "@smithy/fetch-http-handler": "^5.8.0",
++ "@smithy/node-http-handler": "^4.12.1",
++ "@smithy/types": "^4.19.0",
++ "tslib": "^2.6.2"
++ },
++ "engines": {
++ "node": ">=20.0.0"
++ }
++ },
++ "node_modules/@aws-sdk/credential-provider-ini": {
++ "version": "3.973.17",
++ "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-ini/-/credential-provider-ini-3.973.17.tgz",
++ "integrity": "sha512-ged4KXdBkvIC81bLvNHHuQKdKak/VXhQTR1NWYTTqW0474nlmsxy9O/vlgTIohDDWH3xpBdtVMZRyjb+DnocDA==",
++ "license": "Apache-2.0",
++ "dependencies": {
++ "@aws-sdk/core": "^3.978.1",
++ "@aws-sdk/credential-provider-env": "^3.972.72",
++ "@aws-sdk/credential-provider-http": "^3.972.74",
++ "@aws-sdk/credential-provider-login": "^3.972.79",
++ "@aws-sdk/credential-provider-process": "^3.972.72",
++ "@aws-sdk/credential-provider-sso": "^3.973.16",
++ "@aws-sdk/credential-provider-web-identity": "^3.972.78",
++ "@aws-sdk/nested-clients": "^3.997.46",
++ "@aws-sdk/types": "^3.974.6",
++ "@smithy/core": "^3.35.0",
++ "@smithy/credential-provider-imds": "^4.5.2",
++ "@smithy/types": "^4.19.0",
++ "tslib": "^2.6.2"
++ },
++ "engines": {
++ "node": ">=20.0.0"
++ }
++ },
++ "node_modules/@aws-sdk/credential-provider-login": {
++ "version": "3.972.79",
++ "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-login/-/credential-provider-login-3.972.79.tgz",
++ "integrity": "sha512-L+Z85anONJd8MaiuraO4wRxATCdEejBZ3K3eymzWI5JPXa9sOS9CkIm72PBKqXKX+Z9p9NGMX5AIMXm0LEflgw==",
++ "license": "Apache-2.0",
++ "dependencies": {
++ "@aws-sdk/core": "^3.978.1",
++ "@aws-sdk/nested-clients": "^3.997.46",
++ "@aws-sdk/types": "^3.974.6",
++ "@smithy/core": "^3.35.0",
++ "@smithy/types": "^4.19.0",
++ "tslib": "^2.6.2"
++ },
++ "engines": {
++ "node": ">=20.0.0"
++ }
++ },
++ "node_modules/@aws-sdk/credential-provider-node": {
++ "version": "3.972.84",
++ "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-node/-/credential-provider-node-3.972.84.tgz",
++ "integrity": "sha512-oHt854odINVwzwsh+c5x69j0ajm4DbqqqVJ+O1ECsCIZeMDAbzFpXItaqP7UZstJj/ATdTk/KFSH0LaNAgV+kA==",
++ "license": "Apache-2.0",
++ "dependencies": {
++ "@aws-sdk/credential-provider-env": "^3.972.72",
++ "@aws-sdk/credential-provider-http": "^3.972.74",
++ "@aws-sdk/credential-provider-ini": "^3.973.17",
++ "@aws-sdk/credential-provider-process": "^3.972.72",
++ "@aws-sdk/credential-provider-sso": "^3.973.16",
++ "@aws-sdk/credential-provider-web-identity": "^3.972.78",
++ "@aws-sdk/types": "^3.974.6",
++ "@smithy/core": "^3.35.0",
++ "@smithy/credential-provider-imds": "^4.5.2",
++ "@smithy/types": "^4.19.0",
++ "tslib": "^2.6.2"
++ },
++ "engines": {
++ "node": ">=20.0.0"
++ }
++ },
++ "node_modules/@aws-sdk/credential-provider-process": {
++ "version": "3.972.72",
++ "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-process/-/credential-provider-process-3.972.72.tgz",
++ "integrity": "sha512-rLIp2xbMjX/k9/od7APpqq1ZgXXnV0pOL1Th3ZsL8Wu0TRtBsDTVS8iPqcfRFcHakFxPvR04OSTv2ka2qOb/2A==",
++ "license": "Apache-2.0",
++ "dependencies": {
++ "@aws-sdk/core": "^3.978.1",
++ "@aws-sdk/types": "^3.974.6",
++ "@smithy/core": "^3.35.0",
++ "@smithy/types": "^4.19.0",
++ "tslib": "^2.6.2"
++ },
++ "engines": {
++ "node": ">=20.0.0"
++ }
++ },
++ "node_modules/@aws-sdk/credential-provider-sso": {
++ "version": "3.973.16",
++ "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-sso/-/credential-provider-sso-3.973.16.tgz",
++ "integrity": "sha512-IGihaJfFZYacJJr/odqILCoK7W/mvrZ7cuK7ECn3sAu4vLC6u0V8bS7mCGbdugJ8Aum2tnvqmx0F2MRFp2rn9g==",
++ "license": "Apache-2.0",
++ "dependencies": {
++ "@aws-sdk/core": "^3.978.1",
++ "@aws-sdk/nested-clients": "^3.997.46",
++ "@aws-sdk/token-providers": "3.1138.0",
++ "@aws-sdk/types": "^3.974.6",
++ "@smithy/core": "^3.35.0",
++ "@smithy/types": "^4.19.0",
++ "tslib": "^2.6.2"
++ },
++ "engines": {
++ "node": ">=20.0.0"
++ }
++ },
++ "node_modules/@aws-sdk/credential-provider-web-identity": {
++ "version": "3.972.78",
++ "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-web-identity/-/credential-provider-web-identity-3.972.78.tgz",
++ "integrity": "sha512-/y9WvNtlcPBGLR0qc1a+9J/xtYZfVczvLUOuXaVWylzttH7ewsxwHtjmiJSolNrVSDorIxHGHMU61CbonRkmwA==",
++ "license": "Apache-2.0",
++ "dependencies": {
++ "@aws-sdk/core": "^3.978.1",
++ "@aws-sdk/nested-clients": "^3.997.46",
++ "@aws-sdk/types": "^3.974.6",
++ "@smithy/core": "^3.35.0",
++ "@smithy/types": "^4.19.0",
++ "tslib": "^2.6.2"
++ },
++ "engines": {
++ "node": ">=20.0.0"
++ }
++ },
++ "node_modules/@aws-sdk/middleware-sdk-s3": {
++ "version": "3.972.77",
++ "resolved": "https://registry.npmjs.org/@aws-sdk/middleware-sdk-s3/-/middleware-sdk-s3-3.972.77.tgz",
++ "integrity": "sha512-E7W2UOeUoc+lg3uIfR/dM7ZwusHwhBQrKMnlkRv4EXRR+C0YtV1pg25xC7GdZIhXH+NAMgZPCbE7o5to2cjFiw==",
++ "license": "Apache-2.0",
++ "dependencies": {
++ "@aws-sdk/core": "^3.978.1",
++ "@aws-sdk/signature-v4-multi-region": "^3.996.47",
++ "@aws-sdk/types": "^3.974.6",
++ "@smithy/core": "^3.35.0",
++ "@smithy/types": "^4.19.0",
++ "tslib": "^2.6.2"
++ },
++ "engines": {
++ "node": ">=20.0.0"
++ }
++ },
++ "node_modules/@aws-sdk/nested-clients": {
++ "version": "3.997.46",
++ "resolved": "https://registry.npmjs.org/@aws-sdk/nested-clients/-/nested-clients-3.997.46.tgz",
++ "integrity": "sha512-oRxtBcka/JGHGs9l9p9IVajGoTP8vTPmoAzdHGy4Qcy9P5vPnDf6nhIeM/COQNY9k/OahImTRaLkHftoXvfcmQ==",
++ "license": "Apache-2.0",
++ "dependencies": {
++ "@aws-sdk/core": "^3.978.1",
++ "@aws-sdk/signature-v4-multi-region": "^3.996.47",
++ "@aws-sdk/types": "^3.974.6",
++ "@smithy/core": "^3.35.0",
++ "@smithy/fetch-http-handler": "^5.8.0",
++ "@smithy/node-http-handler": "^4.12.1",
++ "@smithy/types": "^4.19.0",
++ "tslib": "^2.6.2"
++ },
++ "engines": {
++ "node": ">=20.0.0"
++ }
++ },
++ "node_modules/@aws-sdk/signature-v4-multi-region": {
++ "version": "3.996.47",
++ "resolved": "https://registry.npmjs.org/@aws-sdk/signature-v4-multi-region/-/signature-v4-multi-region-3.996.47.tgz",
++ "integrity": "sha512-Zk08macMvQTHzQJCLJVkOlviVoqwYMrpXv4lmLN7b7sAbiMoOK7Go0NYdR5UeF+MW8LIbRmwrNy9u/5VvX1U5g==",
++ "license": "Apache-2.0",
++ "dependencies": {
++ "@aws-sdk/types": "^3.974.6",
++ "@smithy/signature-v4": "^5.7.3",
++ "@smithy/types": "^4.19.0",
++ "tslib": "^2.6.2"
++ },
++ "engines": {
++ "node": ">=20.0.0"
++ }
++ },
++ "node_modules/@aws-sdk/token-providers": {
++ "version": "3.1138.0",
++ "resolved": "https://registry.npmjs.org/@aws-sdk/token-providers/-/token-providers-3.1138.0.tgz",
++ "integrity": "sha512-GpyAr0DD63YOEmYFM6Df+gJuIgC92MMTiBK4FTKfxii5MJ9ge20epR7LyroulscYlG89J+ZB2ivFDPjvfQhzdw==",
++ "license": "Apache-2.0",
++ "dependencies": {
++ "@aws-sdk/core": "^3.978.1",
++ "@aws-sdk/nested-clients": "^3.997.46",
++ "@aws-sdk/types": "^3.974.6",
++ "@smithy/core": "^3.35.0",
++ "@smithy/types": "^4.19.0",
++ "tslib": "^2.6.2"
++ },
++ "engines": {
++ "node": ">=20.0.0"
++ }
++ },
++ "node_modules/@aws-sdk/types": {
++ "version": "3.974.6",
++ "resolved": "https://registry.npmjs.org/@aws-sdk/types/-/types-3.974.6.tgz",
++ "integrity": "sha512-v/clNZzZnDxGyvpHMOGpJKVXFAExJzUNAAjaWGdcx8QAcXLGwTaOkw33p5SHAi0YAioK32xB3hWwOekRVfmfKg==",
++ "license": "Apache-2.0",
++ "dependencies": {
++ "@smithy/types": "^4.19.0",
++ "tslib": "^2.6.2"
++ },
++ "engines": {
++ "node": ">=20.0.0"
++ }
++ },
++ "node_modules/@aws-sdk/xml-builder": {
++ "version": "3.972.41",
++ "resolved": "https://registry.npmjs.org/@aws-sdk/xml-builder/-/xml-builder-3.972.41.tgz",
++ "integrity": "sha512-ctjVSyCMegrWfXlx6VqzSBFI6UqmQ5ZlnfMhdLIiWmhoH8UAQxSCP5N3OpG7X3k4LnS7ou74C4mt20+bfTW2aQ==",
++ "license": "Apache-2.0",
++ "dependencies": {
++ "@smithy/types": "^4.19.0",
++ "tslib": "^2.6.2"
++ },
++ "engines": {
++ "node": ">=20.0.0"
++ }
++ },
++ "node_modules/@aws/lambda-invoke-store": {
++ "version": "0.3.0",
++ "resolved": "https://registry.npmjs.org/@aws/lambda-invoke-store/-/lambda-invoke-store-0.3.0.tgz",
++ "integrity": "sha512-sl4Bm6yiMNYrZKkqqDFWN0UfnWhlS8ivKxrYl+6t0gCLrqr8y3B2IqZZbFRkfaVVp7C/baApyh71P+LeE1A2sQ==",
++ "license": "Apache-2.0",
++ "engines": {
++ "node": ">=18.0.0"
++ }
++ },
++ "node_modules/@smithy/core": {
++ "version": "3.35.0",
++ "resolved": "https://registry.npmjs.org/@smithy/core/-/core-3.35.0.tgz",
++ "integrity": "sha512-zRMhfkByhT2snNdr1si24vJitU6Cr9ix2MikUfWmkAgp4jrNP0GcKSP5YvwQ+TlI8AZXER5QOGJn3JsVtSD9/A==",
++ "license": "Apache-2.0",
++ "dependencies": {
++ "@smithy/types": "^4.19.0",
++ "tslib": "^2.6.2"
++ },
++ "engines": {
++ "node": ">=18.0.0"
++ }
++ },
++ "node_modules/@smithy/credential-provider-imds": {
++ "version": "4.5.2",
++ "resolved": "https://registry.npmjs.org/@smithy/credential-provider-imds/-/credential-provider-imds-4.5.2.tgz",
++ "integrity": "sha512-A9uSdn72ozbRUSit0eib0TW7nXuNPlaeM0zcGkJ+nE6tFcSDbnmtwoxbTCFBukVQcszDAyvsd7+rTduPTXpygg==",
++ "license": "Apache-2.0",
++ "dependencies": {
++ "@smithy/core": "^3.33.2",
++ "@smithy/types": "^4.17.2",
++ "tslib": "^2.6.2"
++ },
++ "engines": {
++ "node": ">=18.0.0"
++ }
++ },
++ "node_modules/@smithy/fetch-http-handler": {
++ "version": "5.8.0",
++ "resolved": "https://registry.npmjs.org/@smithy/fetch-http-handler/-/fetch-http-handler-5.8.0.tgz",
++ "integrity": "sha512-ycSJu3tFAQ4v04CBB0agqFMVsSQ1iG3yw+SpgxRqKfaURpQD4CZ8Wn0zPMmSnOuTpTh65Vz+EA0rMrw089wvkA==",
++ "license": "Apache-2.0",
++ "dependencies": {
++ "@smithy/core": "^3.33.3",
++ "@smithy/types": "^4.18.0",
++ "tslib": "^2.6.2"
++ },
++ "engines": {
++ "node": ">=18.0.0"
++ }
++ },
++ "node_modules/@smithy/node-http-handler": {
++ "version": "4.12.1",
++ "resolved": "https://registry.npmjs.org/@smithy/node-http-handler/-/node-http-handler-4.12.1.tgz",
++ "integrity": "sha512-ThMkboGeONWXAelq9FvGsuJC4rOi+qyC4/zhUF58xYpxUg5sQKx2VXZYJmtNjr4dSuBJ1HeJXETQILCz3wOHvw==",
++ "license": "Apache-2.0",
++ "dependencies": {
++ "@smithy/core": "^3.33.3",
++ "@smithy/types": "^4.18.0",
++ "tslib": "^2.6.2"
++ },
++ "engines": {
++ "node": ">=18.0.0"
++ }
++ },
++ "node_modules/@smithy/signature-v4": {
++ "version": "5.7.3",
++ "resolved": "https://registry.npmjs.org/@smithy/signature-v4/-/signature-v4-5.7.3.tgz",
++ "integrity": "sha512-7ImGm+FkHRLcBaRttIAMZ6bzJZWb2cJGoYjq46F2UjycujWzrL9GEN9h4w7eQyXJYnltrUhxbbieBAIRrdqpow==",
++ "license": "Apache-2.0",
++ "dependencies": {
++ "@smithy/core": "^3.33.3",
++ "@smithy/types": "^4.17.2",
++ "tslib": "^2.6.2"
++ },
++ "engines": {
++ "node": ">=18.0.0"
++ }
++ },
++ "node_modules/@smithy/types": {
++ "version": "4.19.0",
++ "resolved": "https://registry.npmjs.org/@smithy/types/-/types-4.19.0.tgz",
++ "integrity": "sha512-r7jh49VJxGerfAcTQA6gXcKc+98zOp/tqRwzYjgOE+iSQsP6cEU1hq2QzbuipmP68QtYdY9wKEhiCQZIzHgZ4Q==",
++ "license": "Apache-2.0",
++ "dependencies": {
++ "tslib": "^2.6.2"
++ },
++ "engines": {
++ "node": ">=18.0.0"
++ }
++ },
"node_modules/accepts": {
"version": "1.3.8",
"resolved": "https://registry.npmjs.org/accepts/-/accepts-1.3.8.tgz",
@@@ -156,6 +546,12 @@
"npm": "1.2.8000 || >= 1.4.16"
}
},
++ "node_modules/bowser": {
++ "version": "2.14.1",
++ "resolved": "https://registry.npmjs.org/bowser/-/bowser-2.14.1.tgz",
++ "integrity": "sha512-tzPjzCxygAKWFOJP011oxFHs57HzIhOEracIgAePE4pqB3LikALKnSzUyU4MGs9/iCEUuHlAJTjTc5M+u7YEGg==",
++ "license": "MIT"
++ },
"node_modules/buffer": {
"version": "5.7.1",
"resolved": "https://registry.npmjs.org/buffer/-/buffer-5.7.1.tgz",
@@@ -1474,6 +1870,12 @@
"node": ">=0.6"
}
},
++ "node_modules/tslib": {
++ "version": "2.8.1",
++ "resolved": "https://registry.npmjs.org/tslib/-/tslib-2.8.1.tgz",
++ "integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==",
++ "license": "0BSD"
++ },
"node_modules/tunnel-agent": {
"version": "0.6.0",
"resolved": "https://registry.npmjs.org/tunnel-agent/-/tunnel-agent-0.6.0.tgz",
ddiff --git a/package.json b/package.json
iindex b48c2c6..c51c81e 100644
---- a/package.json
++++ b/package.json
@@@ -24,6 +24,7 @@
"license": "AGPL-3.0-only",
"repository": "github:OpenVibers/OpenVibe.Host",
"dependencies": {
++ "@aws-sdk/client-s3": "^3.1138.0",
"better-sqlite3": "^11.10.0",
"busboy": "^1.6.0",
"cookie-parser": "^1.4.7",
ddiff --git a/test/backup-offsite.test.js b/test/backup-offsite.test.js
nnew file mode 100644
iindex 0000000..19c8ce8
---- /dev/null
++++ b/test/backup-offsite.test.js
@@@ -0,0 +1,533 @@
++'use strict';
++/**
++ * ovhost backup --all, retention, permissions, the off-host copy (encryption + a mocked S3 client)
++ * and restore-download, all against the fake host. No S3 request leaves the process.
++ */
++const assert = require('assert');
++const crypto = require('crypto');
++const path = require('path');
++const { Readable } = require('stream');
++const { pipeline } = require('stream/promises');
++const { scenario, test, runTests, SECRET } = require('./helpers');
++const { main } = require('../lib/cli');
++const bcrypto = require('../lib/backup-crypto');
++const retention = require('../lib/retention');
++const offsite = require('../lib/offsite');
++const { normalise } = require('../lib/inventory');
++
++const DAY = 86400000;
++const KEY_HEX = crypto.createHash('sha256').update('test backup key').digest('hex');
++const S3_SECRET = `${SECRET}-s3`;
++
++/** An in-memory S3 that records every command it is sent. */
++function mockS3({ pageSize = 1000 } = {}) {
++ const objects = new Map();
++ const uploads = new Map();
++ const sent = [];
++ let seq = 0;
++ const s3 = {
++ objects,
++ sent,
++ failPut: null, // (key) -> true to fail that PutObject/UploadPart
++ async send(cmd) {
++ const name = cmd.constructor.name;
++ const i = cmd.input;
++ sent.push({ name, input: { ...i, Body: i.Body ? `<${Buffer.byteLength(i.Body)} bytes>` : undefined } });
++ switch (name) {
++ case 'PutObjectCommand':
++ if (s3.failPut && s3.failPut(i.Key)) throw new Error('503 SlowDown');
++ assert.strictEqual(i.ContentLength, i.Body.length);
++ objects.set(i.Key, Buffer.from(i.Body));
++ return { ETag: '"put"' };
++ case 'CreateMultipartUploadCommand': { const id = `up-${++seq}`; uploads.set(id, { key: i.Key, parts: new Map() }); return { UploadId: id }; }
++ case 'UploadPartCommand':
++ if (s3.failPut && s3.failPut(i.Key)) throw new Error('503 SlowDown');
++ uploads.get(i.UploadId).parts.set(i.PartNumber, Buffer.from(i.Body));
++ return { ETag: `"p${i.PartNumber}"` };
++ case 'CompleteMultipartUploadCommand': {
++ const u = uploads.get(i.UploadId);
++ const nums = i.MultipartUpload.Parts.map((p) => p.PartNumber);
++ assert.deepStrictEqual(nums, [...u.parts.keys()].sort((a, b) => a - b));
++ objects.set(u.key, Buffer.concat(nums.map((n) => u.parts.get(n))));
++ uploads.delete(i.UploadId);
++ return {};
++ }
++ case 'AbortMultipartUploadCommand': uploads.delete(i.UploadId); return {};
++ case 'ListObjectsV2Command': {
++ const keys = [...objects.keys()].filter((k) => k.startsWith(i.Prefix || '')).sort();
++ const start = i.ContinuationToken ? Number(i.ContinuationToken) : 0;
++ const size = Math.min(pageSize, i.MaxKeys || 1000);
++ const page = keys.slice(start, start + size);
++ const more = start + size < keys.length;
++ return { Contents: page.map((k) => ({ Key: k, Size: objects.get(k).length })), IsTruncated: more, NextContinuationToken: more ? String(start + size) : undefined };
++ }
++ case 'GetObjectCommand': {
++ const o = objects.get(i.Key);
++ if (!o) throw Object.assign(new Error('NoSuchKey'), { name: 'NoSuchKey' });
++ const pieces = [];
++ for (let k = 0; k < o.length; k += 1000) pieces.push(o.subarray(k, k + 1000));
++ return { Body: Readable.from(pieces) };
++ }
++ case 'DeleteObjectCommand': objects.delete(i.Key); return {};
++ default: throw new Error(`mock S3: unexpected ${name}`);
++ }
++ },
++ };
++ return s3;
++}
++
++/** The standard fake host plus AI (two databases) and OpenRe (not deployed), a backup env and key. */
++function backupHost({ envMode = 0o600, keyMode = 0o600, envOwner = 'root', env = null } = {}) {
++ const host = scenario();
++ const doc = JSON.parse(host.read('/etc/openvibe/host.json'));
++ doc.services.ai = { repo: '/opt/openvibe.ai', units: [], databases: [{ name: 'ai', path: '/var/lib/openvibe-ai/ai.db' }, { name: 'ai-extra', path: '/var/lib/openvibe-ai/extra.db' }] };
++ doc.services.openre = { repo: '/opt/openre', units: [], databases: [{ name: 'openre', path: '/var/lib/openre/openre.db' }] };
++ host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });
++ host.inv = normalise(doc);
++ host.put('/var/lib/openvibe-ai/ai.db', 'sqlite', { owner: 'ubuntu' });
++ host.put('/var/lib/openvibe-ai/extra.db', 'sqlite', { owner: 'ubuntu' });
++ host.put('/etc/openvibe/backup.env', env || [
++ '# off-host backups',
++ 'BACKUP_S3_ENDPOINT=https://s3.us-west-004.backblazeb2.com',
++ 'BACKUP_S3_BUCKET=openvibe-backups-test',
++ 'BACKUP_S3_PREFIX=openvibe-backups',
++ 'BACKUP_S3_KEY_ID=004keyid0000000000000001',
++ `BACKUP_S3_SECRET="${S3_SECRET}"`,
++ 'BACKUP_ENCRYPTION_KEY_FILE=/etc/openvibe/backup.key',
++ '',
++ ].join('\n'), { mode: envMode, owner: envOwner });
++ host.put('/etc/openvibe/backup.key', `${KEY_HEX}\n`, { mode: keyMode, owner: 'root' });
++ // Distinct content per database, so a restore can be compared byte for byte.
++ host.backupContent = (db) => Buffer.from(`SQLite format 3\0 copy of ${db} ${'x'.repeat(5000)}`);
++ host.sqliteHandler = (db, sql) => (/quick_check/.test(sql) ? [{ quick_check: 'ok' }] : [{ n: 0 }]);
++ host.s3 = mockS3();
++ host.s3Configs = [];
++ host.cli = async (...argv) => {
++ const lines = [];
++ const code = await main(argv, { exec: host.exec, out: (s) => lines.push(s), env: {}, s3: (cfg) => { host.s3Configs.push(cfg); return host.s3; } });
++ return { code, out: lines.join('\n') };
++ };
++ return host;
++}
++
++function dirsOf(host, dir) {
++ return [...host.files.keys()].filter((k) => path.dirname(k) === dir).map((k) => path.basename(k)).sort();
++}
++
++function noSecrets(text, what) {
++ for (const s of [S3_SECRET, SECRET, KEY_HEX]) assert.ok(!String(text).includes(s), `${what} leaked a secret value`);
++}
++
++async function decrypt(key, buf) {
++ const chunks = [];
++ await pipeline(Readable.from([buf]), bcrypto.createDecryptStream(key), async (src) => { for await (const c of src) chunks.push(c); });
++ return Buffer.concat(chunks);
++}
++
++async function encrypt(key, buf, opts) {
++ const chunks = [];
++ const pieces = [];
++ for (let i = 0; i < buf.length; i += 777) pieces.push(buf.subarray(i, i + 777));
++ await pipeline(Readable.from(pieces), bcrypto.createEncryptStream(key, opts), async (src) => { for await (const c of src) chunks.push(c); });
++ return Buffer.concat(chunks);
++}
++
++runTests([
++ // ── encryption ─────────────────────────────────────────────────────────────
++ test('encryption: round trip at every chunk boundary; sizes match encryptedSize()', async () => {
++ const key = crypto.randomBytes(32);
++ const C = 4096;
++ for (const n of [0, 1, C - 1, C, C + 1, 3 * C, 3 * C + 17]) {
++ const plain = crypto.randomBytes(n);
++ const enc = await encrypt(key, plain, { chunkSize: C });
++ assert.strictEqual(enc.length, bcrypto.encryptedSize(n, C), `size for ${n}`);
++ assert.strictEqual(enc.subarray(0, 8).toString(), 'OVBKAES1');
++ assert.ok(n < 64 || !enc.includes(plain.subarray(0, 64)), 'plaintext visible in ciphertext');
++ assert.ok((await decrypt(key, enc)).equals(plain), `round trip for ${n}`);
++ }
++ // Two encryptions of the same file differ (a fresh salt per file).
++ const p = Buffer.from('same');
++ assert.ok(!(await encrypt(key, p)).equals(await encrypt(key, p)));
++ }),
++
++ test('encryption: tampering, truncation, appended data, a header edit and another key all fail', async () => {
++ const key = crypto.randomBytes(32);
++ const C = 1024;
++ const plain = crypto.randomBytes(3 * C + 100);
++ const enc = await encrypt(key, plain, { chunkSize: C });
++ const flip = Buffer.from(enc); flip[bcrypto.HEADER_LEN + C + 5] ^= 1;
++ await assert.rejects(decrypt(key, flip), /failed authentication/);
++ // Cut exactly after the second whole chunk: the last one left lacks the final flag.
++ await assert.rejects(decrypt(key, enc.subarray(0, bcrypto.HEADER_LEN + 2 * (C + 16))), /failed authentication/);
++ await assert.rejects(decrypt(key, enc.subarray(0, enc.length - 1)), /failed authentication/);
++ await assert.rejects(decrypt(key, enc.subarray(0, 20)), /truncated/);
++ await assert.rejects(decrypt(key, Buffer.concat([enc, enc.subarray(bcrypto.HEADER_LEN)])), /failed authentication/);
++ const hdr = Buffer.from(enc); hdr[20] ^= 1; // the salt
++ await assert.rejects(decrypt(key, hdr), /failed authentication/);
++ await assert.rejects(decrypt(crypto.randomBytes(32), enc), /another key/);
++ assert.ok((await decrypt(key, enc)).equals(plain));
++ }),
++
++ test('encryption key file: 64 hex characters or base64; errors never echo the text', () => {
++ const k = crypto.randomBytes(32);
++ assert.ok(bcrypto.parseKey(`${k.toString('hex')}\n`).equals(k));
++ assert.ok(bcrypto.parseKey(k.toString('base64')).equals(k));
++ assert.throws(() => bcrypto.parseKey('hunter2-not-a-key'), (err) => !err.message.includes('hunter2') && /64 hex/.test(err.message));
++ assert.match(bcrypto.keyId(k), /^[0-9a-f]{16}$/);
++ }),
++
++ // ── retention ──────────────────────────────────────────────────────────────
++ test('retention policy: newest per day for 7 days, newest per ISO week for 4 weeks', () => {
++ const stamps = [];
++ for (let d = 0; d < 60; d++) {
++ const t = Date.parse('2026-08-01T03:30:00Z') + d * DAY;
++ stamps.push(new Date(t).toISOString().replace(/[-:]/g, '').replace('T', '-').slice(0, 15));
++ stamps.push(new Date(t + 3600000).toISOString().replace(/[-:]/g, '').replace('T', '-').slice(0, 15)); // a second backup that day
++ }
++ const kept = [...retention.keep(stamps, { daily: 7, weekly: 4 })].sort();
++ // Last day 2026-09-29 (Tue). Days 09-23..09-29; weeks W40 (09-29), W39 (09-27), W38 (09-20), W37 (09-13).
++ assert.deepStrictEqual(kept, ['20260913-043000', '20260920-043000', '20260923-043000', '20260924-043000', '20260925-043000', '20260926-043000', '20260927-043000', '20260928-043000', '20260929-043000']);
++ assert.strictEqual(retention.isoWeek(Date.parse('2021-01-03T00:00:00Z')), '2020-W53');
++ assert.deepStrictEqual([...retention.keep(['junk', '20260101-000000'])], ['20260101-000000']);
++ }),
++
++ test('backup --all prunes to 7 daily + 4 weekly per service; failing days never age out good backups', async () => {
++ const host = backupHost();
++ // A directory ovhost did not record is never touched.
++ host.put('/var/backups/openvibe/live/20200101-000000/live.db', 'operator copy', { owner: 'root', mode: 0o600 });
++ for (let d = 0; d < 40; d++) {
++ const r = await host.cli('backup', '--all');
++ assert.strictEqual(r.code, 0, r.out);
++ host.advance(DAY);
++ }
++ // 40 daily runs from 2026-09-22 12:00 to 2026-10-31 12:00.
++ const expected = ['20200101-000000', '20261011-120000', '20261018-120000', '20261025-120000', '20261026-120000', '20261027-120000', '20261028-120000', '20261029-120000', '20261030-120000', '20261031-120000'];
++ assert.deepStrictEqual(dirsOf(host, '/var/backups/openvibe/live'), expected);
++ assert.deepStrictEqual(dirsOf(host, '/var/backups/openvibe/media'), expected.slice(1));
++
++ // Live fails for 20 days: nothing of Live is pruned, the others carry on.
++ host.onSqliteBackup = (db) => (db.includes('openvibe.live') ? new Error('database disk image is malformed') : null);
++ for (let d = 0; d < 20; d++) {
++ const r = await host.cli('backup', '--all');
++ assert.strictEqual(r.code, 2);
++ host.advance(DAY);
++ }
++ assert.deepStrictEqual(dirsOf(host, '/var/backups/openvibe/live'), expected);
++ assert.strictEqual(dirsOf(host, '/var/backups/openvibe/media').length, 9);
++ assert.ok(dirsOf(host, '/var/backups/openvibe/media').includes('20261119-120000'));
++ }),
++
++ test('a partly failed backup is kept until a newer good one exists, then pruned; drills skip it', async () => {
++ const host = backupHost();
++ await host.cli('backup', '--all');
++ host.advance(DAY);
++ host.onSqliteBackup = (db) => (db.endsWith('extra.db') ? new Error('disk I/O error') : null);
++ const bad = await host.cli('backup', '--all', '--json');
++ assert.strictEqual(bad.code, 2);
++ const s = JSON.parse(bad.out);
++ const ai = s.services.find((x) => x.service === 'ai');
++ assert.strictEqual(ai.status, 'failed');
++ assert.match(ai.error, /ai-extra: disk I\/O error/);
++ assert.deepStrictEqual(dirsOf(host, ai.dir), ['ai.db'], 'the good half is kept');
++ const log = host.read('/var/lib/openvibe-host/backups/ai.jsonl').trim().split('\n').map((l) => JSON.parse(l));
++ assert.strictEqual(log[log.length - 1].ok, false);
++ assert.deepStrictEqual(dirsOf(host, '/var/backups/openvibe/ai'), ['20260922-120000', '20260923-120000']);
++ host.onSqliteBackup = null;
++ host.advance(DAY);
++ await host.cli('backup', '--all');
++ assert.deepStrictEqual(dirsOf(host, '/var/backups/openvibe/ai'), ['20260922-120000', '20260924-120000'], 'the failed one went once a newer good one existed');
++ }),
++
++ // ── failure isolation and the summary ─────────────────────────────────────
++ test('one failing service never stops the others; exit code 2; JSON summary per run', async () => {
++ const host = backupHost();
++ host.onSqliteBackup = (db) => (db.includes('openvibe.media') ? new Error('SQLITE_BUSY: database is locked') : null);
++ const r = await host.cli('backup', '--all');
++ assert.strictEqual(r.code, 2, r.out);
++ assert.match(r.out, /FAILED\s+media\s+media: SQLITE_BUSY/);
++ assert.match(r.out, /OK\s+live\s+1 file\(s\)/);
++ assert.match(r.out, /OK\s+ai\s+2 file\(s\)/);
++ assert.match(r.out, /SKIPPED\s+openre\s+no database found/);
++ const summary = JSON.parse(host.read('/var/lib/openvibe-host/backup-runs/20260922-120000.json'));
++ assert.strictEqual(summary.ok, false);
++ assert.deepStrictEqual(summary.services.map((x) => [x.service, x.status]), [['live', 'ok'], ['media', 'failed'], ['ai', 'ok'], ['openre', 'skipped']]);
++ assert.deepStrictEqual(summary.retention, { daily: 7, weekly: 4 });
++ // The services after the failing one were still backed up.
++ assert.ok(host.files.has('/var/backups/openvibe/ai/20260922-120000/ai-extra.db'));
++ assert.strictEqual(host.files.has('/var/backups/openvibe/openre'), true);
++ assert.deepStrictEqual(dirsOf(host, '/var/backups/openvibe/openre'), [], 'no empty backup directory is left behind');
++ assert.strictEqual(host.files.has('/var/lib/openvibe-host/locks/_backup-all.lock'), false, 'lock released');
++ const ok = await (host.advance(1000), host.cli('backup', '--all', '--json'));
++ host.onSqliteBackup = null;
++ assert.strictEqual(JSON.parse(ok.out).ok, false);
++ host.advance(1000);
++ assert.strictEqual((await host.cli('backup', '--all')).code, 0);
++ }),
++
++ test('permissions: backup directories root 0700, copies root 0600, no -wal/-shm; old service-owned directories are locked down', async () => {
++ const host = backupHost();
++ host.ensureDir('/var/backups/openvibe/live', 'ubuntu', 0o750);
++ // The worker leaves a side file behind: it is not kept.
++ host.onSqliteBackup = (db, dest) => { host.put(`${dest}-wal`, 'wal', { owner: 'ubuntu', mode: 0o644 }); return null; };
++ const r = await host.cli('backup', '--all', '--offsite');
++ assert.strictEqual(r.code, 0, r.out);
++ let dirs = 0;
++ let files = 0;
++ for (const [p, e] of host.files) {
++ if (p !== '/var/backups/openvibe' && !p.startsWith('/var/backups/openvibe/')) continue;
++ if (e.type === 'dir') { dirs += 1; assert.deepStrictEqual([e.owner, e.mode], ['root', 0o700], p); }
++ else { files += 1; assert.deepStrictEqual([e.owner, e.mode], ['root', 0o600], p); assert.ok(!/-(wal|shm|journal)$/.test(p), `${p} left behind`); }
++ }
++ assert.ok(dirs >= 8 && files === 4, `${dirs} dirs, ${files} files`);
++ assert.deepStrictEqual(dirsOf(host, '/var/backups/openvibe.staging'), [], 'staging emptied');
++ assert.deepStrictEqual([host.files.get('/var/backups/openvibe.staging').owner, host.files.get('/var/backups/openvibe.staging').mode], ['root', 0o711]);
++ // The worker still ran as the service user, into a directory it owned at the time.
++ assert.ok(host.sqliteCalls.filter((c) => c.op === 'backup').every((c) => c.as === 'ubuntu' && c.dest.startsWith('/var/backups/openvibe.staging/')));
++
++ // restore-download: a root-only directory, 0600 files.
++ const d = await host.cli('restore-download', 'ai', 'latest');
++ assert.strictEqual(d.code, 0, d.out);
++ for (const p of ['/var/lib/openvibe-restore', '/var/lib/openvibe-restore/ai-20260922-120000']) assert.deepStrictEqual([host.files.get(p).owner, host.files.get(p).mode], ['root', 0o700], p);
++ for (const f of ['ai.db', 'ai-extra.db']) {
++ const e = host.files.get(`/var/lib/openvibe-restore/ai-20260922-120000/${f}`);
++ assert.deepStrictEqual([e.owner, e.mode], ['root', 0o600], f);
++ }
++ }),
++
++ test('a copy that is a link is refused, not chowned', async () => {
++ const host = backupHost();
++ // The service user hard-links its "copy" to some other file before root takes it over.
++ const orig = host.exec.sqliteBackup;
++ host.exec.sqliteBackup = async (db, dest, o) => { await orig(db, dest, o); host.files.get(dest).nlink = 2; };
++ const r = await host.cli('backup', 'live');
++ host.exec.sqliteBackup = orig;
++ assert.strictEqual(r.code, 2);
++ assert.match(r.out, /not a plain file/);
++ assert.ok(!host.calls.some((c) => (c.cmd === 'chown' || c.cmd === 'chmod') && String(c.args[c.args.length - 1]).endsWith('live.db')), 'a linked copy is never chowned');
++ assert.deepStrictEqual(dirsOf(host, '/var/backups/openvibe/live'), [], 'and never moved into the backups');
++ }),
++
++ // ── off-host upload ────────────────────────────────────────────────────────
++ test('--offsite: every copy encrypted and uploaded; manifest with HMAC; no secret anywhere', async () => {
++ const host = backupHost();
++ const r = await host.cli('backup', '--all', '--offsite');
++ assert.strictEqual(r.code, 0, r.out);
++ const base = 'openvibe-backups/fake-host/20260922-120000/';
++ assert.deepStrictEqual([...host.s3.objects.keys()].sort(), [`${base}ai/ai-extra.db.ovbk`, `${base}ai/ai.db.ovbk`, `${base}live/live.db.ovbk`, `${base}manifest.json`, `${base}media/media.db.ovbk`]);
++ const key = Buffer.from(KEY_HEX, 'hex');
++ for (const [k, v] of host.s3.objects) {
++ if (k.endsWith('manifest.json')) continue;
++ assert.ok(!v.includes('copy of'), `${k} is not encrypted`);
++ const plain = await decrypt(key, v);
++ const svc = k.split('/')[3];
++ const file = path.basename(k, '.ovbk');
++ assert.ok(plain.equals(host.files.get(`/var/backups/openvibe/${svc}/20260922-120000/${file}`).content), `${k} round trip`);
++ }
++ const m = JSON.parse(host.s3.objects.get(`${base}manifest.json`));
++ assert.strictEqual(m.hmac, bcrypto.hmacManifest(key, JSON.stringify((({ hmac, ...rest }) => rest)(m))));
++ assert.strictEqual(m.keyId, bcrypto.keyId(key));
++ assert.deepStrictEqual(Object.keys(m.services), ['live', 'media', 'ai']);
++ // Only PutObject for small files; path-style B2 endpoint and region from the endpoint.
++ assert.ok(host.s3.sent.filter((c) => /Put/.test(c.name)).every((c) => c.input.Bucket === 'openvibe-backups-test'));
++ const cfg = host.s3Configs[0];
++ assert.deepStrictEqual([cfg.endpoint, cfg.region, cfg.forcePathStyle, cfg.credentials.accessKeyId], ['https://s3.us-west-004.backblazeb2.com', 'us-west-004', true, '004keyid0000000000000001']);
++ assert.strictEqual(cfg.credentials.secretAccessKey, S3_SECRET);
++ noSecrets(JSON.stringify(cfg), 'the config object');
++ noSecrets(r.out, 'CLI output');
++ noSecrets(host.read('/var/lib/openvibe-host/backup-runs/20260922-120000.json'), 'the run summary');
++ noSecrets(JSON.stringify(host.s3.sent), 'S3 requests');
++ noSecrets(host.s3.objects.get(`${base}manifest.json`), 'the manifest');
++ const summary = JSON.parse(host.read('/var/lib/openvibe-host/backup-runs/20260922-120000.json'));
++ assert.strictEqual(summary.offsite.ok, true);
++ assert.strictEqual(summary.offsite.uploaded.length, 4);
++ assert.strictEqual(summary.offsite.location, base);
++ }),
++
++ test('large files go up as a multipart upload; a failed part aborts it', async () => {
++ const host = backupHost();
++ const big = crypto.randomBytes(300 * 1024);
++ host.backupContent = (db) => (db.includes('openvibe.live') ? big : Buffer.from('small'));
++ await host.cli('backup', '--all');
++ const summary = await require('../lib/commands/backup').readRun(host.exec, host.inv, null);
++ const cfg = await offsite.loadConfig(host.exec, {});
++ const ctx = { exec: host.exec, inv: host.inv, log: () => {} };
++ const res = await offsite.push(ctx, summary, { cfg, client: host.s3, partSize: 64 * 1024, chunkSize: 16 * 1024 });
++ assert.strictEqual(res.ok, true);
++ const live = res.uploaded.find((u) => u.service === 'live');
++ assert.strictEqual(live.parts, Math.ceil(live.cipherBytes / (64 * 1024)));
++ const partSizes = host.s3.sent.filter((c) => c.name === 'UploadPartCommand').map((c) => c.input.Body);
++ assert.ok(partSizes.slice(0, -1).every((b) => b === `<${64 * 1024} bytes>`), 'every part but the last is exactly partSize');
++ assert.strictEqual(live.cipherBytes, bcrypto.encryptedSize(big.length, 16 * 1024));
++ assert.ok((await decrypt(Buffer.from(KEY_HEX, 'hex'), host.s3.objects.get(live.object))).equals(big));
++ assert.ok(host.s3.sent.some((c) => c.name === 'CompleteMultipartUploadCommand'));
++
++ const s3 = mockS3();
++ let n = 0;
++ s3.failPut = () => ++n === 3;
++ await assert.rejects(offsite.uploadIterable(s3, 'b', 'k', [crypto.randomBytes(200 * 1024)], { partSize: 64 * 1024 }), /SlowDown/);
++ assert.ok(s3.sent.some((c) => c.name === 'AbortMultipartUploadCommand'));
++ assert.strictEqual(s3.objects.size, 0);
++ }),
++
++ test('off-host retention: runs older than 30 days are deleted, the newest 7 always stay, and nothing is pruned after a failed upload', async () => {
++ const host = backupHost();
++ const put = (run) => {
++ host.s3.objects.set(`openvibe-backups/fake-host/${run}/live/live.db.ovbk`, Buffer.from('x'));
++ host.s3.objects.set(`openvibe-backups/fake-host/${run}/manifest.json`, Buffer.from('{}'));
++ };
++ // Ten runs 35..44 days old, one 5 days old; another host's and a stray object are never touched.
++ for (let d = 35; d < 45; d++) put(require('../lib/commands/backup').stamp(Date.parse('2026-09-22T12:00:00Z') - d * DAY));
++ put('20260917-120000');
++ host.s3.objects.set('openvibe-backups/other-host/20200101-000000/manifest.json', Buffer.from('{}'));
++ host.s3.objects.set('openvibe-backups/fake-host/README', Buffer.from('keep'));
++
++ // A failed upload: no pruning at all.
++ host.s3.failPut = (k) => k.endsWith('media.db.ovbk');
++ const bad = await host.cli('backup', '--all', '--offsite', '--json');
++ assert.strictEqual(bad.code, 2);
++ const badSummary = JSON.parse(bad.out);
++ assert.strictEqual(badSummary.offsite.ok, false);
++ assert.match(badSummary.offsite.failures[0].error, /SlowDown/);
++ assert.deepStrictEqual(badSummary.offsite.pruned, []);
++ // Eleven old runs, README, the other host's run and this partial run: all still there.
++ assert.strictEqual(new Set([...host.s3.objects.keys()].map((k) => k.split('/')[2])).size, 14);
++
++ // offsite push re-uploads that run; now it is complete and old runs go.
++ host.s3.failPut = null;
++ const again = await host.cli('offsite', 'push', '--run', '20260922-120000');
++ assert.strictEqual(again.code, 0, again.out);
++ const runs = [...new Set([...host.s3.objects.keys()].filter((k) => k.startsWith('openvibe-backups/fake-host/2')).map((k) => k.split('/')[2]))].sort();
++ // Newest 7 with a manifest: this run, 09-17, and the five youngest of the old ones (35..39 days).
++ assert.deepStrictEqual(runs, ['20260814-120000', '20260815-120000', '20260816-120000', '20260817-120000', '20260818-120000', '20260917-120000', '20260922-120000']);
++ assert.ok(host.s3.objects.has('openvibe-backups/other-host/20200101-000000/manifest.json'));
++ assert.ok(host.s3.objects.has('openvibe-backups/fake-host/README'));
++ const summary = JSON.parse(host.read('/var/lib/openvibe-host/backup-runs/20260922-120000.json'));
++ assert.strictEqual(summary.offsite.ok, true);
++ assert.strictEqual(summary.offsite.pruned.length, 5);
++ }),
++
++ test('offsite list and check; paginated listings', async () => {
++ const host = backupHost();
++ host.s3 = mockS3({ pageSize: 2 });
++ await host.cli('backup', '--all', '--offsite');
++ host.advance(DAY);
++ await host.cli('backup', '--all', '--offsite');
++ const l = await host.cli('offsite', 'list', 'live', '--json');
++ assert.strictEqual(l.code, 0, l.out);
++ assert.deepStrictEqual(JSON.parse(l.out).map((x) => [x.run, x.objects, x.manifest]), [['20260923-120000', 5, true], ['20260922-120000', 5, true]]);
++ const c = await host.cli('offsite', 'check');
++ assert.strictEqual(c.code, 0, c.out);
++ assert.match(c.out, /s3:\/\/openvibe-backups-test\/openvibe-backups\/fake-host\/ .*region us-west-004.*key id [0-9a-f]{16}; retention 30 days/);
++ noSecrets(c.out, 'offsite check');
++ }),
++
++ // ── restore-download ───────────────────────────────────────────────────────
++ test('restore-download: verified, decrypted copies in a new directory; never overwrites; never near a live database', async () => {
++ const host = backupHost();
++ await host.cli('backup', '--all', '--offsite');
++ const original = host.files.get('/var/backups/openvibe/live/20260922-120000/live.db').content;
++ const r = await host.cli('restore-download', 'live', '20260922-120000');
++ assert.strictEqual(r.code, 0, r.out);
++ const dest = '/var/lib/openvibe-restore/live-20260922-120000/live.db';
++ assert.ok(host.files.get(dest).content.equals(original));
++ assert.match(r.out, /quick_check ok/);
++ assert.ok(host.sqliteCalls.some((c) => c.db === dest && /quick_check/.test(c.sql)));
++ // The live database was never written.
++ assert.strictEqual(host.read('/opt/openvibe.live/data/live.db'), 'sqlite');
++
++ const again = await host.cli('restore-download', 'live', '20260922-120000');
++ assert.strictEqual(again.code, 1);
++ assert.match(again.out, /not empty; restore-download never overwrites/);
++ for (const out of ['/opt/openvibe.live/data', '/opt/openvibe.live', '/opt', '/var/lib/openvibe-ai', '/var/backups/openvibe/live/x']) {
++ const bad = await host.cli('restore-download', 'live', 'latest', '--out', out);
++ assert.strictEqual(bad.code, 1, out);
++ assert.match(bad.out, /refusing --out/, out);
++ }
++ const other = await host.cli('restore-download', 'live', 'latest', '--out', '/root/restore-test', '--json');
++ assert.strictEqual(other.code, 0, other.out);
++ assert.strictEqual(JSON.parse(other.out).files[0].sha256, crypto.createHash('sha256').update(original).digest('hex'));
++ }),
++
++ test('restore-download refuses a tampered object, a forged manifest and a run that lacks the service', async () => {
++ const host = backupHost();
++ await host.cli('backup', '--all', '--offsite');
++ const base = 'openvibe-backups/fake-host/20260922-120000/';
++ const obj = Buffer.from(host.s3.objects.get(`${base}media/media.db.ovbk`));
++ obj[obj.length - 20] ^= 1;
++ host.s3.objects.set(`${base}media/media.db.ovbk`, obj);
++ const t = await host.cli('restore-download', 'media', 'latest');
++ assert.strictEqual(t.code, 2);
++ assert.match(t.out, /failed authentication/);
++ assert.strictEqual(host.files.has('/var/lib/openvibe-restore/media-20260922-120000/media.db'), false, 'a bad copy is removed');
++
++ // Swapping two encrypted files is caught by the manifest's hashes.
++ host.s3.objects.set(`${base}ai/ai.db.ovbk`, host.s3.objects.get(`${base}ai/ai-extra.db.ovbk`));
++ const sw = await host.cli('restore-download', 'ai', 'latest', '--out', '/root/r-ai');
++ assert.strictEqual(sw.code, 2);
++ assert.match(sw.out, /does not match the manifest/);
++
++ const m = JSON.parse(host.s3.objects.get(`${base}manifest.json`));
++ m.services.live.files[0].plainSha256 = '0'.repeat(64);
++ host.s3.objects.set(`${base}manifest.json`, Buffer.from(JSON.stringify(m)));
++ const f = await host.cli('restore-download', 'live', 'latest', '--out', '/root/r-live');
++ assert.strictEqual(f.code, 2);
++ assert.match(f.out, /failed its HMAC check/);
++
++ const none = await host.cli('restore-download', 'openre', 'latest');
++ assert.strictEqual(none.code, 1);
++ assert.match(none.out, /no off-host run of openre/);
++ }),
++
++ // ── configuration ──────────────────────────────────────────────────────────
++ test('config: root-only env file and key; missing names listed by name; not root refused; no value ever printed', async () => {
++ let host = backupHost({ envMode: 0o644 });
++ let r = await host.cli('offsite', 'check');
++ assert.strictEqual(r.code, 1);
++ assert.match(r.out, /backup env file \/etc\/openvibe\/backup.env must not be readable or writable by group or others/);
++
++ host = backupHost({ envOwner: 'ubuntu' });
++ r = await host.cli('offsite', 'check');
++ assert.match(r.out, /must be owned by root/);
++
++ host = backupHost({ keyMode: 0o640 });
++ r = await host.cli('offsite', 'check');
++ assert.strictEqual(r.code, 1);
++ assert.match(r.out, /encryption key file \/etc\/openvibe\/backup.key must not be readable/);
++
++ host = backupHost({ env: `BACKUP_S3_ENDPOINT=https://s3.us-west-004.backblazeb2.com\nBACKUP_S3_SECRET=${S3_SECRET}\nBACKUP_S3_KEY_ID=\n` });
++ r = await host.cli('offsite', 'check');
++ assert.strictEqual(r.code, 1);
++ assert.match(r.out, /missing or has empty: BACKUP_S3_BUCKET, BACKUP_S3_KEY_ID, BACKUP_ENCRYPTION_KEY_FILE/);
++ noSecrets(r.out, 'a config error');
++
++ host = backupHost({ env: `BACKUP_S3_ENDPOINT=http://${S3_SECRET}.example.com\nBACKUP_S3_BUCKET=b-ok\nBACKUP_S3_KEY_ID=k\nBACKUP_S3_SECRET=${S3_SECRET}\nBACKUP_ENCRYPTION_KEY_FILE=/etc/openvibe/backup.key\n` });
++ r = await host.cli('offsite', 'check');
++ assert.match(r.out, /must be https/);
++ noSecrets(r.out, 'an endpoint error');
++
++ // A broken off-host config still leaves every local backup made, and fails the run.
++ host = backupHost({ keyMode: 0o644 });
++ r = await host.cli('backup', '--all', '--offsite');
++ assert.strictEqual(r.code, 2);
++ assert.match(r.out, /OFFSITE\s+FAILED: encryption key file/);
++ assert.ok(host.files.has('/var/backups/openvibe/live/20260922-120000/live.db'));
++
++ const { createFakeHost } = require('./fake-host');
++ const plain = createFakeHost({ root: false, user: 'ubuntu' });
++ const doc = JSON.parse(host.read('/etc/openvibe/host.json'));
++ plain.put('/etc/openvibe/host.json', JSON.stringify(doc), { owner: 'ubuntu' });
++ const lines = [];
++ const code = await main(['restore-download', 'live', 'latest'], { exec: plain.exec, out: (s) => lines.push(s), env: {}, s3: () => mockS3() });
++ assert.strictEqual(code, 1);
++ assert.match(lines.join('\n'), /run as root/);
++ }),
++
++ test('usage: backup --all takes no service; restore-download needs a run; bad --run is refused', async () => {
++ const host = backupHost();
++ assert.strictEqual((await host.cli('backup', 'live', '--all')).code, 1);
++ assert.strictEqual((await host.cli('restore-download', 'live')).code, 1);
++ assert.strictEqual((await host.cli('offsite', 'push', '--run', '../etc')).code, 2);
++ assert.strictEqual((await host.cli('offsite', 'push')).code, 1, 'no run recorded yet');
++ assert.strictEqual((await host.cli('backup', '--all', '--keep-daily', '0')).code, 1);
++ }),
++]);
ddiff --git a/test/deploy.test.js b/test/deploy.test.js
iindex e2f0655..eae80d0 100644
---- a/test/deploy.test.js
++++ b/test/deploy.test.js
@@@ -271,7 +271,9 @@ runTests([
const b = host.sqliteCalls.find((c) => c.op === 'backup');
assert.ok(b, 'backup taken');
assert.strictEqual(b.as, 'ubuntu');
-- assert.match(b.dest, /^\/var\/backups\/openvibe\/live\/\d{8}-\d{6}\/live\.db$/);
++ assert.match(b.dest, /^\/var\/backups\/openvibe\.staging\/live-\d{8}-\d{6}\/live\.db$/, 'the worker writes to its staging directory');
++ const dir = [...host.files.keys()].find((k) => /^\/var\/backups\/openvibe\/live\/\d{8}-\d{6}\/live\.db$/.test(k));
++ assert.ok(dir, 'the copy ends up under /var/backups/openvibe/live/<stamp>/');
const backupIdx = host.calls.findIndex((c) => c.cmd === 'sqlite-backup');
const restartIdx = host.calls.findIndex((c) => c.cmd === 'systemctl' && c.args[0] === 'restart');
assert.ok(backupIdx < restartIdx, 'backup before restart');
ddiff --git a/test/drill.test.js b/test/drill.test.js
iindex f298e96..a5da90a 100644
---- a/test/drill.test.js
++++ b/test/drill.test.js
@@@ -155,9 +155,10 @@ runTests([
const calls = host.calls.slice(callsBefore);
// The copy is made as the service user; integrity and counts run as the service user.
-- const cp = calls.find((c) => c.cmd === 'cp');
-- assert.strictEqual(cp.as, 'ubuntu');
-- assert.deepStrictEqual(cp.args.slice(-2), [`${host.backupDir}/community.db`, `${tmp}/db/community.db`]);
++ // Backups are root-only: root copies, and the copy belongs to the service user, 0600.
++ const cp = calls.find((c) => c.cmd === 'install' && c.args.includes(`${tmp}/db/community.db`));
++ assert.strictEqual(cp.privileged, true);
++ assert.deepStrictEqual(cp.args, ['-o', 'ubuntu', '-m', '0600', '-T', '--', `${host.backupDir}/community.db`, `${tmp}/db/community.db`]);
const integrity = host.sqliteCalls.find((c) => /integrity_check/.test(c.sql));
assert.deepStrictEqual([integrity.db, integrity.as], [`${tmp}/db/community.db`, 'ubuntu']);
@@@ -368,7 +369,7 @@ runTests([
host = await drillScenario({ backup: false });
r = await host.cli('drill', 'community');
assert.strictEqual(r.code, 1);
-- assert.match(r.out, /no ovhost backup recorded for community/);
++ assert.match(r.out, /no good ovhost backup recorded for community/);
host = await drillScenario({ backup: false });
host.ensureDir('/srv/old-backup');
ddiff --git a/test/executor.test.js b/test/executor.test.js
iindex 102ec69..8a6442f 100644
---- a/test/executor.test.js
++++ b/test/executor.test.js
@@@ -45,6 +45,24 @@ runTests([
db.close();
}),
++ test('sqlite backup of a WAL database: one 0600 file in rollback-journal mode, no -wal/-shm left', async () => {
++ const src = path.join(dir, 'wal-src.db');
++ const db = new Database(src);
++ db.pragma('journal_mode = WAL');
++ db.exec('CREATE TABLE t (x); INSERT INTO t VALUES (1), (2), (3);');
++ const out = fs.mkdtempSync(path.join(dir, 'bk-'));
++ const dest = path.join(out, 'wal.db');
++ await exec.sqliteBackup(src, dest);
++ db.close();
++ assert.deepStrictEqual(fs.readdirSync(out), ['wal.db']);
++ assert.strictEqual(fs.statSync(dest).mode & 0o777, 0o600);
++ const copy = new Database(dest, { readonly: true });
++ assert.strictEqual(copy.pragma('journal_mode', { simple: true }), 'delete');
++ assert.strictEqual(copy.prepare('SELECT count(*) AS n FROM t').get().n, 3);
++ copy.close();
++ assert.deepStrictEqual(fs.readdirSync(out), ['wal.db'], 'reading the copy leaves no side files');
++ }),
++
test('drill primitives: integrity_check through the worker, kill() signals exactly one pid', async () => {
const dbPath = path.join(dir, 'restored.db');
const db = new Database(dbPath);
ddiff --git a/test/fake-host.js b/test/fake-host.js
iindex 2ccaed5..97ab420 100644
---- a/test/fake-host.js
++++ b/test/fake-host.js
@@@ -7,6 +7,7 @@
*/
const crypto = require('crypto');
const path = require('path');
++const { Readable, Writable } = require('stream');
function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', start = Date.parse('2026-09-22T12:00:00Z') } = {}) {
const files = new Map(); // path -> { type: 'file'|'dir'|'symlink', content, mode, owner, target }
@@@ -31,6 +32,7 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', st
onKill: null, // (pid, signal) -> false to ignore the signal
systemdRuns: [],
nextPid: 40000,
++ streamPiece: 4096, // readStream() yields files in pieces of this size
};
// ── filesystem ──
@@@ -225,7 +227,41 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', st
case 'systemctl': return systemctlCmd(args);
case 'nginx': return { stdout: '', ...host.nginxTest() };
case 'rm': rmTree(args[args.length - 1]); return { code: 0, stdout: '', stderr: '' };
-- case 'install': { const src = args[args.length - 2]; const dest = args[args.length - 1]; put(dest, host.read(src)); return { code: 0, stdout: '', stderr: '' }; }
++ case 'install': {
++ const src = args[args.length - 2];
++ const dest = args[args.length - 1];
++ const o = args.indexOf('-o');
++ const m = args.indexOf('-m');
++ const e = get(src);
++ if (!e || e.type !== 'file') return { code: 1, stdout: '', stderr: `install: cannot stat '${src}': No such file or directory` };
++ put(dest, e.content, { owner: o >= 0 ? args[o + 1] : 'root', mode: m >= 0 ? parseInt(args[m + 1], 8) : 0o755 });
++ return { code: 0, stdout: '', stderr: '' };
++ }
++ case 'chown': {
++ const e = files.get(path.resolve(args[args.length - 1]));
++ if (!e) return { code: 1, stdout: '', stderr: 'chown: no such file' };
++ e.owner = args.filter((a) => !a.startsWith('-'))[0].split(':')[0];
++ return { code: 0, stdout: '', stderr: '' };
++ }
++ case 'chmod': {
++ const e = files.get(path.resolve(args[args.length - 1]));
++ if (!e) return { code: 1, stdout: '', stderr: 'chmod: no such file' };
++ e.mode = parseInt(args.filter((a) => !a.startsWith('-'))[0], 8);
++ return { code: 0, stdout: '', stderr: '' };
++ }
++ case 'mv': {
++ const src = path.resolve(args[args.length - 2]);
++ const dest = path.resolve(args[args.length - 1]);
++ if (!files.has(src)) return { code: 1, stdout: '', stderr: `mv: cannot stat '${src}'` };
++ if (files.has(dest)) return { code: 1, stdout: '', stderr: `mv: '${dest}' exists` };
++ ensureDir(path.dirname(dest));
++ for (const k of [...files.keys()]) {
++ if (k !== src && !k.startsWith(`${src}/`)) continue;
++ files.set(dest + k.slice(src.length), files.get(k));
++ files.delete(k);
++ }
++ return { code: 0, stdout: '', stderr: '' };
++ }
case 'node': return { code: 0, stdout: '', stderr: '' };
case 'systemd-run': return systemdRunCmd(args);
case 'cp': {
@@@ -241,6 +277,29 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', st
},
async readFile(p) { host.reads.push(path.resolve(p)); const e = get(p); return e && e.type === 'file' ? (Buffer.isBuffer(e.content) ? e.content.toString('utf8') : String(e.content)) : null; },
async writeFile(p, content, { mode = 0o644 } = {}) { put(p, content, { mode, owner: user }); },
++ readStream(p) {
++ const e = get(p);
++ if (!e || e.type !== 'file') {
++ const r = new Readable({ read() {} });
++ process.nextTick(() => r.destroy(Object.assign(new Error(`ENOENT: no such file or directory, open '${p}'`), { code: 'ENOENT' })));
++ return r;
++ }
++ host.reads.push(path.resolve(p));
++ const buf = Buffer.isBuffer(e.content) ? e.content : Buffer.from(String(e.content));
++ // Small pieces, so chunking and part boundaries are exercised.
++ const pieces = [];
++ for (let i = 0; i < buf.length; i += host.streamPiece) pieces.push(buf.subarray(i, i + host.streamPiece));
++ return Readable.from(pieces);
++ },
++ writeStream(p, { mode = 0o600 } = {}) {
++ if (files.has(path.resolve(p))) throw Object.assign(new Error(`EEXIST: file already exists, open '${p}'`), { code: 'EEXIST' });
++ const parts = [];
++ put(p, Buffer.alloc(0), { mode, owner: user });
++ return new Writable({
++ write(chunk, _enc, cb) { parts.push(Buffer.from(chunk)); cb(); },
++ final(cb) { put(p, Buffer.concat(parts), { mode, owner: user }); cb(); },
++ });
++ },
async appendFile(p, text) { const prev = host.read(p) || ''; put(p, prev + text, { mode: 0o640, owner: user }); },
async removeFile(p) { files.delete(path.resolve(p)); },
async symlink(target, p) { ensureDir(path.dirname(p)); files.set(path.resolve(p), { type: 'symlink', target, owner: user, mode: 0o777 }); },
@@@ -250,7 +309,7 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', st
const e = get(p);
if (!e) return null;
const size = e.type === 'file' ? Buffer.byteLength(Buffer.isBuffer(e.content) ? e.content : String(e.content)) : 4096;
-- return { mode: e.mode, uid: e.owner === 'root' ? 0 : 1000, gid: 0, owner: e.owner, isFile: e.type === 'file', isDir: e.type === 'dir', isSymlink: !!l && l.type === 'symlink', size };
++ return { mode: e.mode, uid: e.owner === 'root' ? 0 : 1000, gid: 0, owner: e.owner, isFile: e.type === 'file', isDir: e.type === 'dir', isSymlink: !!l && l.type === 'symlink', size, nlink: e.nlink || 1 };
},
async readdir(p) {
const abs = path.resolve(p);
@@@ -283,7 +342,10 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', st
host.sqliteCalls.push({ op: 'backup', db, dest, as });
calls.push({ cmd: 'sqlite-backup', args: [db, dest], as: as || null, privileged: false, cwd: null });
if (files.has(path.resolve(dest))) throw new Error(`refusing to overwrite ${dest}`);
-- put(dest, `backup of ${db}`, { owner: as || user, mode: 0o640 });
++ const parent = files.get(path.dirname(path.resolve(dest)));
++ if (as && parent && parent.owner !== as) throw new Error(`cannot open ${dest}: permission denied (directory owned by ${parent.owner}, worker runs as ${as})`);
++ if (host.onSqliteBackup) { const r = host.onSqliteBackup(db, dest, as); if (r instanceof Error) throw r; }
++ put(dest, host.backupContent ? host.backupContent(db) : `backup of ${db}`, { owner: as || user, mode: 0o600 });
},
async listeners(port) { return listeners.get(port) || []; },
async kill(pid, signal = 'SIGTERM') {
ddiff --git a/test/snapshot-backup.test.js b/test/snapshot-backup.test.js
iindex aa20511..4097ffd 100644
---- a/test/snapshot-backup.test.js
++++ b/test/snapshot-backup.test.js
@@@ -45,7 +45,13 @@ runTests([
const call = host.sqliteCalls.find((c) => c.op === 'backup');
assert.strictEqual(call.as, 'ubuntu');
assert.strictEqual(call.db, '/opt/openvibe.media/data/media.db');
-- assert.strictEqual(host.files.get(res.dir).owner, 'ubuntu', 'backup dir owned by the service user');
++ assert.strictEqual(call.dest, '/var/backups/openvibe.staging/media-20260922-120000/media.db', 'the worker writes into its staging directory');
++ // Backups hold user data: root:root, directories 0700, files 0600.
++ for (const d of ['/var/backups/openvibe', '/var/backups/openvibe/media', res.dir]) {
++ assert.deepStrictEqual([host.files.get(d).owner, host.files.get(d).mode], ['root', 0o700], d);
++ }
++ assert.deepStrictEqual([host.files.get(res.files[0].dest).owner, host.files.get(res.files[0].dest).mode], ['root', 0o600]);
++ assert.strictEqual(host.files.has('/var/backups/openvibe.staging/media-20260922-120000'), false, 'staging directory removed');
const log = host.read('/var/lib/openvibe-host/backups/media.jsonl');
assert.strictEqual(JSON.parse(log.trim()).files[0].dest, res.files[0].dest);
// A second backup in the same second never overwrites the first.