openvibers/openvibe.host Commits

Restore drills for live and media: both inventory entries are now drill-supported, relying on the products' own switches (LIVE_DRILL from OpenVibe.Live 8a58aea, MEDIA_DRILL from OpenVibe.Media 6e74eb0; drill.requires refuses an older checkout). live runs on 13000 with the copy as DB_PATH and DATA_DIR={tmp}/data, compares /api/themes, /api/emotes/global, /api/streams, /api/streams/recently-online?limit=20 and /api/streams/channel/japaneseoldguy/live (ignoring what changes while someone is live, and the one Media-sourced thumbnail) and counts users, channels, managed_streams, streams, follows and chat_messages. media runs on 14100 with the copy as DB_PATH and its storage paths under {tmp}/storage (never created; THUMBNAILS_PATH stays production's, read-only, for the /browse tab counts), compares /browse?tab=videos and /browse?tab=clips byte for byte and counts media_objects, vods, clips and apps. test/drill-live-media.test.js runs both drills from host.example.json on the fake host. docs/restore-drills.md describes both (the results table is unchanged: neither has run on the host), and backups.md, storage-state.md and the README follow

OP OpenVibers committed 23 Sept 2026 · eaa226ccca · 6 files, +357 −15


Changes

ddiff --git a/README.md b/README.md
iindex 0f6bb10..f6e1251 100644
---- a/README.md
++++ b/README.md
@@@ -151,7 +151,7 @@ Each inventory entry's `drill` block declares:
- `counts`: `{ db, table }` pairs;
- `supported: false` with a `reason` when a second instance cannot run without side effects.
--In `host.example.json`, 24 services have drill blocks, and 21 can be drilled. live, media and ai are marked unsupported, and the reasons are in the file. For a service with several units, `unit` (as tools does: only the docs app runs) or `command` (an absolute argv, as openre does: only `openre-api` runs) says which one process the drill starts. `productionPort` says which production port to compare against, `databases` entries can be data directories (`{ "env": "DATA_DIR", "dir": true }`), `bind` redirects paths a service opens relative to its checkout into the drill directory (`BindPaths=`, in the drill unit's own mount namespace), and `requires` refuses the drill unless the deployed checkout has each switch the overrides rely on. The Tools and Games drills rely only on switches those repositories already have. [docs/restore-drills.md](docs/restore-drills.md) lists what each repository could add to make its drill stricter.
++In `host.example.json`, 24 services have drill blocks, and 23 can be drilled. ai is marked unsupported, and the reason is in the file. live and media rely on their own drill switches (`LIVE_DRILL`, `MEDIA_DRILL`), which `requires` checks for in the deployed checkout. For a service with several units, `unit` (as tools does: only the docs app runs) or `command` (an absolute argv, as openre does: only `openre-api` runs) says which one process the drill starts. `productionPort` says which production port to compare against, `databases` entries can be data directories (`{ "env": "DATA_DIR", "dir": true }`), `bind` redirects paths a service opens relative to its checkout into the drill directory (`BindPaths=`, in the drill unit's own mount namespace), and `requires` refuses the drill unless the deployed checkout has each switch the overrides rely on. The Tools and Games drills rely only on switches those repositories already have. [docs/restore-drills.md](docs/restore-drills.md) lists what each repository could add to make its drill stricter.
The sandbox blocks writes outside the drill directory and addresses beyond loopback. Loopback stays open, so the overrides are what keep a drill away from production services. Common overrides are `OV_OAUTH_CLIENT_SECRET=` (no service tokens) and `http://127.0.0.1:9` for URLs whose empty value would fall back to a production service.
ddiff --git a/docs/backups.md b/docs/backups.md
iindex 79b2a7f..c94b906 100644
---- a/docs/backups.md
++++ b/docs/backups.md
@@@ -350,8 +350,9 @@ The drill:
Record the result in [restore-drills.md](restore-drills.md) and note "off-host" in the Backup column.
--For services without a drill (live, media, ai), download the copy and check it by
--hand:
++For a service without a drill (ai), or for live and media until the release with their drill
++switch is deployed (`LIVE_DRILL`, `MEDIA_DRILL`; `drill.requires` refuses an older checkout), download the
++copy and check it by hand:
```
sudo ovhost restore-download live latest --out /var/lib/openvibe-restore/check-live
ddiff --git a/docs/restore-drills.md b/docs/restore-drills.md
iindex f6248f6..a0fab0f 100644
---- a/docs/restore-drills.md
++++ b/docs/restore-drills.md
@@@ -139,6 +139,50 @@ Supported with declared overrides (the table above records which have passed on
stay out of every ovhost command: they are `workerUnits`, which are listed and never restarted.
Run it after the OpenRe release with `OPENRE_DRILL` is deployed (OpenRe `docs/cutover.md`
phase C).
++- live (not run yet). Needs OpenVibe.Live `8a58aea` or later; `drill.requires` refuses an older
++ checkout. The production unit's ExecStart starts with `LIVE_DRILL=1`, `DB_PATH` on the copy and
++ `DATA_DIR={tmp}/data`. In that mode (Live `server/drill.js`, `docs/deploy.md#restore-drills`) Live
++ refuses to start unless `DB_PATH` and `DATA_DIR` are outside the checkout and `/opt/openvibe.live`,
++ `HOST` is loopback, `PORT` is not 3000 and no socket was handed over by systemd. It writes only
++ under `DATA_DIR` (a fresh `analytics.db`, empty data directories; the per-location `*_PATH` values
++ in `live.env` are ignored) and the copy. It starts only its HTTP server:
++ - no RTMP, SRT, WHIP, SFU or JSMPEG listener, and no TURN credential;
++ - no WebSocket: upgrades get 403;
++ - no jobs loop, restream or relay resume, AI job, Media reconciler, Events outbox, chat bridge,
++ identity sync, deploy notice, star job or registry refresh.
++
++ Every method but GET, HEAD and OPTIONS answers 403. Outbound connections, `fetch`, programs other
++ than `git`, UDP sockets and other listeners are refused in-process, so Media, Community and
++ Network look down to it. `rs-companion.db` is not restored: the server never opens it. Compared:
++ - `/api/themes` and `/api/emotes/global`, byte for byte;
++ - `/api/streams` and `/api/streams/channel/japaneseoldguy/live`, without the values production
++ changes while someone is live (viewer counts, heartbeats, live thumbnails, recording state);
++ - `/api/streams/recently-online?limit=20`, without `vod_thumbnail`, the one value it asks Media for.
++
++ Counts: `users`, `channels`, `managed_streams`, `streams`, `follows`, `chat_messages`. Not
++ compared: `/release.json`, which production computes at its own boot, so a `public/`-only deploy
++ without a restart makes it differ; and the home statistics, which are cached for 30 s and windowed
++ by the clock. Any existing channel can replace `japaneseoldguy`.
++- media (not run yet). Needs OpenVibe.Media `6e74eb0` or later (`drill.requires`). `MEDIA_DRILL=1`
++ with `DB_PATH` on the copy. Media then refuses to start unless `DB_PATH` is outside the checkout
++ and `/opt/openvibe.media`, `HOST` is loopback and `PORT` is not 4100. It starts only its HTTP
++ server:
++ - no app seeding or JWKS refresh;
++ - no tiering sweep, health job, junk sweep, clip re-cuts, copy verification, jobs worker, invariant
++ scans, disk guardian, thumbnail cleanup, object purge, backfill or orphan-recording finalize;
++ - no Events relay and no webhook (the `apps` table's URLs are production's).
++
++ It writes no file and creates no directory. It never opens a file path from the database: every
++ byte route (`/v` and `/c` bytes, `/t`, `/a`, `/f`, `/o`, paste screenshots, live frames) answers
++ 503, while watch pages and `/browse` render from the copy. Every method but GET, HEAD and OPTIONS
++ answers 403 and `/auth/*` 503. Connections, `fetch`, programs other than `git` (no ffmpeg or
++ ffprobe), UDP and other listeners are refused in-process. `/api/ready` drops the storage and
++ remote-tier checks and reports `"mode": "drill"`. The storage paths from `media.env` are moved
++ under `{tmp}/storage`, which is never created, as a second switch. `THUMBNAILS_PATH` stays
++ production's, read-only in the sandbox, because `/browse` counts that directory in its tab bar;
++ the drill lists it and serves none of it. Compared byte for byte: `/browse?tab=videos` and
++ `/browse?tab=clips`. Counts: `media_objects`, `vods`, `clips`, `apps`. Production caches the tab
++ counts for 60 s, so take the backup right before the drill.
Services that are not deployed yet (everything except network, live, media, tools, community and
games as of 22 Sep) have entries built from their repositories' `deploy/` directories.
@@@ -217,9 +261,13 @@ side-effect-free.
Marked `drill: { supported: false }`, with the reason in the inventory:
--- **live:** `data/analytics.db` is opened relative to the checkout at boot. It is read-only in the
-- sandbox, and without the sandbox it would be production's file. RTMP always binds. Restreams, relays,
-- AI jobs and Media reconcilers resume from rows in the copy.
--- **media:** webhooks to Live take their URL from the `apps` table and have no env switch. The health
-- job, junk sweep and clip jobs run ffmpeg against production files through absolute paths stored in
-- the database.
++- **ai:** a drill must start AI with every provider key blanked, or queued and new runs would call
++ paid providers. Backups work.
++
++Until 23 Sep 2026 live and media were unsupported too:
++- **live:** `data/analytics.db` was opened relative to the checkout at boot. RTMP always bound.
++ Restreams, relays, AI jobs and Media reconcilers resumed from rows in the copy.
++- **media:** webhooks to Live took their URL from the `apps` table. The health job, junk sweep and
++ clip jobs ran ffmpeg against production files through absolute paths stored in the database.
++
++`LIVE_DRILL` and `MEDIA_DRILL` (above) replace those reasons.
ddiff --git a/docs/storage-state.md b/docs/storage-state.md
iindex dd58584..85798ff 100644
---- a/docs/storage-state.md
++++ b/docs/storage-state.md
@@@ -94,8 +94,8 @@ log win.
| Service | Backing store (production path; env, default) | Canonical? | ADR-007 status | Readiness checks (handler) | Store checked? | Backup / drill |
|---|---|---|---|---|---|---|
| **Network** | `/opt/openvibe.network/data/network.db`; `DB_PATH`, default `./data/network.db` (`server/config.js:55`). Analytics lives in the same file (`README.md:306-307`). | Yes: identity, OAuth clients, subjects, OpenCoins wallets, themes, notifications, analytics | SQLite, the reviewed single-host state. `docs/platform-plan.md:74,111` defers PostgreSQL "until a room needs it". | `/api/ready` (`server/observability.js:58-96`). Required: `db` (`SELECT COUNT(*) FROM oauth_clients`), `signing_key` in production (signs and verifies an RS256 probe; fails on the HS256 dev key). Optional: `registry_poll`, `discord_bot`. | **Yes** | Drill passed 05:22. ovhost's deploy readiness polls the static `/api/health` (finding 3). |
--| **Live** | `/opt/openvibe.live/data/live.db` (legacy in-place layout, "current production", Live `docs/deploy.md:31`); `DB_PATH`, default `./data/live.db` (`server/config.js:228`). Also `data/rs-companion.db`, and `data/analytics.db`, whose path is hard-coded with no env (`server/index.js:361-363`). | Yes: streams, channels, follows, Live accounts, restreams, **money** (`BILLING_AUTHORITY` unset = `live`), AI state (`vod_ai_state`/`clip_ai_state`), analytics. **Not canonical:** the 12 chat mirror tables (Chat is, since 02:03 UTC) and the frozen `vods`/`clips`/`pastes` (Media and Community are). | SQLite, the reviewed state. No PostgreSQL plan. Money moves to Billing, not to PostgreSQL. | `/api/ready` (`server/index.js:748-756`, `server/web/observability.js:79-106`). Required: `boot`, `db` (`SELECT 1` on `live.db`). Optional: `sfu`, `media` (`MEDIA_URL/healthz`, cached 30 s), `network_key` (PEM loaded, no call). | **Yes** for `live.db`; not `analytics.db` or `rs-companion.db` | `ovhost backup` covers `live.db` and `rs-companion.db`. `analytics.db` is left out on purpose (`host.example.json` `_databasesNote`). Drill **unsupported**: `analytics.db` is opened relative to the checkout, RTMP always binds, and restreams, relays and AI jobs resume from the copy. |
--| **Media** | `/opt/openvibe.media/data/media.db`; `DB_PATH`, default `./data/media.db` (`server/config.js:24`). Local `VOD_PATH`, `CLIPS_PATH`, `PASTES_PATH`, `THUMBNAILS_PATH`, `FILES_PATH`, `OBJECTS_PATH` (`config.js:27-58`), plus B2 and R2 object storage. | Yes: media objects, VODs, clips, files, thumbnails, avatars. Paste writes are frozen (Community is canonical). Media still serves paste screenshot bytes. | SQLite, the reviewed state. No PostgreSQL plan. | `/api/ready` (`server/observability.js:72-107`). Required: `db` (`SELECT COUNT(*) FROM apps`), and `storage_{vods,clips,pastes,thumbnails,files,objects}` (writes and unlinks a probe). Optional: `network_jwks`, `remote_b2`/`remote_r2` (HeadBucket, cached 60 s), `events_outbox` (>1000 pending). Also reports `recordings_in_progress`. | **Yes** (database and disk; object storage optional) | `ovhost backup` covers `media.db`. Drill **unsupported**: webhooks to Live come from the `apps` table, and ffmpeg jobs run against production paths. ovhost's deploy readiness polls `/healthz` (finding 3). |
++| **Live** | `/opt/openvibe.live/data/live.db` (legacy in-place layout, "current production", Live `docs/deploy.md:31`); `DB_PATH`, default `./data/live.db` (`server/config.js:228`). Also `data/rs-companion.db`, and `data/analytics.db`, whose path was hard-coded with no env (`server/index.js:361-363`); from Live `8a58aea` it is `<DATA_DIR>/analytics.db` (`ANALYTICS_DB_PATH` overrides it; `server/paths.js`). | Yes: streams, channels, follows, Live accounts, restreams, **money** (`BILLING_AUTHORITY` unset = `live`), AI state (`vod_ai_state`/`clip_ai_state`), analytics. **Not canonical:** the 12 chat mirror tables (Chat is, since 02:03 UTC) and the frozen `vods`/`clips`/`pastes` (Media and Community are). | SQLite, the reviewed state. No PostgreSQL plan. Money moves to Billing, not to PostgreSQL. | `/api/ready` (`server/index.js:748-756`, `server/web/observability.js:79-106`). Required: `boot`, `db` (`SELECT 1` on `live.db`). Optional: `sfu`, `media` (`MEDIA_URL/healthz`, cached 30 s), `network_key` (PEM loaded, no call). | **Yes** for `live.db`; not `analytics.db` or `rs-companion.db` | `ovhost backup` covers `live.db` and `rs-companion.db`. `analytics.db` is left out on purpose (`host.example.json` `_databasesNote`). Drill **supported** from Live `8a58aea` (`LIVE_DRILL=1`: reads only, files under `DATA_DIR`, no listeners but HTTP, no jobs; not run yet). Before that release it was unsupported: `analytics.db` was opened relative to the checkout, RTMP always bound, and restreams, relays and AI jobs resumed from the copy. |
++| **Media** | `/opt/openvibe.media/data/media.db`; `DB_PATH`, default `./data/media.db` (`server/config.js:24`). Local `VOD_PATH`, `CLIPS_PATH`, `PASTES_PATH`, `THUMBNAILS_PATH`, `FILES_PATH`, `OBJECTS_PATH` (`config.js:27-58`), plus B2 and R2 object storage. | Yes: media objects, VODs, clips, files, thumbnails, avatars. Paste writes are frozen (Community is canonical). Media still serves paste screenshot bytes. | SQLite, the reviewed state. No PostgreSQL plan. | `/api/ready` (`server/observability.js:72-107`). Required: `db` (`SELECT COUNT(*) FROM apps`), and `storage_{vods,clips,pastes,thumbnails,files,objects}` (writes and unlinks a probe). Optional: `network_jwks`, `remote_b2`/`remote_r2` (HeadBucket, cached 60 s), `events_outbox` (>1000 pending). Also reports `recordings_in_progress`. | **Yes** (database and disk; object storage optional) | `ovhost backup` covers `media.db`. Drill **supported** from Media `6e74eb0` (`MEDIA_DRILL=1`: reads only, no webhooks, jobs or ffmpeg, byte routes 503; not run yet). Before that release it was unsupported: webhooks to Live come from the `apps` table, and ffmpeg jobs run against production paths. ovhost's deploy readiness polls `/healthz` (finding 3). |
| **Tools** | The gateway has no database. Each satellite has `apps/<app>/data/analytics.db` (relative to `/opt/openvibe.tools`). img, audio and docs also have `data/jobs.db` (`apps/_shared/jobs/index.js:77`), `data/uploads` and `data/output`; yt has `data/downloads`. Job results go to Media when `TOOLS_JOB_RESULTS=media` (default `local`, `apps/_shared/jobs/index.js:32-36`). | Yes: jobs (img, audio, docs) and per-app analytics | SQLite, the reviewed state. One file per process, so ADR-007's "concurrent writers across processes" trigger does not apply. | `/api/ready` on every unit (`apps/_shared/observe.js:71-162`). Gateway: `catalog` required; `network_key`, `service_directory`, `community`, and each `satellite_<name>` optional (`apps/gateway/server/index.js:60-72`). img, docs, audio: `jobs_db` (`SELECT COUNT(*) FROM tool_jobs`), `job_runtime` and data directories required. yt: `downloads_dir`. food: `maps` upstream. maps, text: none required (`analytics_db` optional). | **Partly**: `jobs.db` yes on its satellites; analytics optional by design; the gateway reports satellites as optional (finding 2) | `ovhost backup` covers the seven `analytics.db` files only; **`jobs.db` is missing** (finding 5). Drill **unsupported**: eight units, relative `analytics.db` paths, job runtimes re-queue at start. |
| **Games** | `/opt/openvibe.games/data/world.db`; `DB_PATH`, default `data/world.db` (`apps/server/src/config.ts:79`). No `.env.example`. | Yes: world, accounts, inventory | SQLite, the reviewed state. No PostgreSQL plan. | `/api/ready` (since `f11e21c`): required `db` (`meta.schema_version` in `world.db`) and `tick` (last step < 5 s ago); 404 on the public vhosts. `/healthz` unchanged. | **Yes** | `ovhost backup` covers `world.db`. Drill **unsupported** (runtime side effects not reviewed). Games is `managed: false` in the inventory. |
| **Community** | `/var/lib/openvibe-community/community.db` (`StateDirectory`, `deploy/systemd/openvibe-community.service:26-27`). `/opt/openvibe.community/data/community.db` is the legacy location, and the inventory keeps whichever of the two exists. `COMMUNITY_DB_PATH`, default `./data/community.db` (`server/config.js:56`). | Yes: pastes (`PASTES_AUTHORITY=community` since 2026-09-22), comments, spaces, threads, Pulse | SQLite, the reviewed state. | `/api/ready` (`server/observability.js:29-66`). Required: `db` (`SELECT COUNT(*) FROM sqlite_master WHERE type='table'`, fails on 0). Optional: `network_jwks`, `media` (community mode) or `live` (live mode). Reports `pastes_authority`. | **Yes** | Drill passed 03:44 (manual) and 05:22. |
ddiff --git a/host.example.json b/host.example.json
iindex 6f93575..004a917 100644
---- a/host.example.json
++++ b/host.example.json
@@@ -77,8 +77,30 @@
"backupOnChange": ["server/db/migrations.js", "server/db/schema.sql"],
"nginx": { "vhost": "openvibe.live.conf", "repoVhost": "deploy/nginx/openvibe.live.conf", "variant": "websocket", "wsPaths": ["/ws/"], "maxBody": "3m" },
"drill": {
-- "supported": false,
-- "reason": "server/index.js opens data/analytics.db relative to the checkout (not movable by env) at module load: in the sandbox the checkout is read-only and boot fails, and without the sandbox the drill would write production's analytics database. It also always binds RTMP (RTMP_PORT and RTMP_PORT+8000), resumes restreams/RobotStreamer relays and AI jobs for rows in the copy, and runs Media reconcilers; several have no env switch. Needs a BindPaths over data/ plus DB-level switches before it can be drilled."
++ "_notes": "Needs OpenVibe.Live 8a58aea or later (LIVE_DRILL, server/drill.js; requires checks the deployed checkout). LIVE_DRILL=1 refuses to start unless DB_PATH and DATA_DIR are outside the checkout and /opt/openvibe.live, HOST is loopback and PORT is not 3000, and never takes over the systemd socket. It then serves HTTP only: no RTMP/SRT/WHIP/SFU/JSMPEG listener, no WebSocket (upgrades get 403), no TURN credential, no jobs loop, restream or relay resume, AI job, Media reconciler, Events outbox, chat bridge, identity sync, deploy notice, star job or registry refresh. Every method but GET/HEAD/OPTIONS answers 403; net, fetch, child_process (git only), dgram and listen are guarded, so every other service looks down to it. All its files go to DATA_DIR ({tmp}/data: a fresh analytics.db and empty data directories; the per-location *_PATH values in live.env are ignored in a drill). rs-companion.db is not restored: the server never opens it. Compared: DB-only public reads; ignore lists drop what production changes in memory or every few seconds while someone is live (viewer counts, heartbeats, live thumbnails, recording state) and the one Media lookup (recently-online vod_thumbnail). /api/streams/channel/japaneseoldguy/live is the star channel: any channel that exists works. /release.json is not compared: production computes it at its own boot, so a public/-only deploy without a restart would make it differ. EVENTS_URL, the client secret and MEDIA_URL are blanked as a second switch. Not run on the host yet.",
++ "port": 13000,
++ "databases": { "live": "DB_PATH" },
++ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "LIVE_DRILL": "1", "DATA_DIR": "{tmp}/data", "EVENTS_URL": "", "OV_OAUTH_CLIENT_SECRET": "", "MEDIA_URL": "http://127.0.0.1:9" },
++ "dirs": ["{tmp}/data"],
++ "requires": [
++ { "file": "server/drill.js", "contains": "LIVE_DRILL" },
++ { "file": "server/paths.js", "contains": "DATA_DIR" }
++ ],
++ "compare": [
++ "/api/themes",
++ "/api/emotes/global",
++ { "path": "/api/streams", "ignore": ["viewer_count", "peak_viewers", "external_viewer_count", "total_viewer_count", "last_heartbeat", "thumbnail_url"] },
++ { "path": "/api/streams/recently-online?limit=20", "ignore": ["vod_thumbnail"] },
++ { "path": "/api/streams/channel/japaneseoldguy/live", "ignore": ["viewer_count", "peak_viewers", "last_heartbeat", "thumbnail_url", "server_clip", "endpoint"] }
++ ],
++ "counts": [
++ { "db": "live", "table": "users" },
++ { "db": "live", "table": "channels" },
++ { "db": "live", "table": "managed_streams" },
++ { "db": "live", "table": "streams" },
++ { "db": "live", "table": "follows" },
++ { "db": "live", "table": "chat_messages" }
++ ]
}
},
"media": {
@@@ -100,8 +122,36 @@
"databases": [{ "name": "media", "path": "/opt/openvibe.media/data/media.db" }],
"nginx": { "vhost": "openvibe.media.conf", "repoVhost": "deploy/nginx/openvibe.media.conf", "variant": "http" },
"drill": {
-- "supported": false,
-- "reason": "Webhooks to Live (orphan-recording finalize at boot, clip re-cuts, storage alerts) read their URL from the apps table and have no env switch; the health job, junk sweep and clip jobs run ffmpeg against production files through absolute paths stored in the database; the tiering sweep and health job start on a timer with no switch. A side-effect-free instance needs the copy edited (webhook_url NULL, storage_tier off) first, which ovhost does not do."
++ "_notes": "Needs OpenVibe.Media 6e74eb0 or later (MEDIA_DRILL, server/drill.js; requires checks the deployed checkout). MEDIA_DRILL=1 refuses to start unless DB_PATH is outside the checkout and /opt/openvibe.media, HOST is loopback and PORT is not 4100. It then serves HTTP only: no app seeding or JWKS refresh, no tiering sweep, health job, junk sweep, clip re-cuts, copy verification, jobs worker, invariant scans, disk guardian, thumbnail cleanup, object purge, backfill, orphan-recording finalize or Events relay, and no webhook (the apps table's URLs are production's). It writes no file and creates no directory, and never opens a file path from the database: every byte route (/v and /c bytes, /t, /a, /f, /o, paste screenshots, live frames) answers 503; watch pages and /browse render from the copy. Every method but GET/HEAD/OPTIONS answers 403, /auth/* 503; net, fetch, child_process (git only: no ffmpeg or ffprobe), dgram and listen are guarded. /api/ready drops the storage and remote-tier checks (mode: drill). The storage paths from media.env move under {tmp}/storage (never created) as a second switch; THUMBNAILS_PATH stays production's, read-only in the sandbox, because /browse counts the thumbnail directory in its tab bar (the drill lists it and serves none of it). /browse caches its tab counts for 60 s in production: take the backup right before the drill. EVENTS_URL and the client secret are blanked too. Not run on the host yet.",
++ "port": 14100,
++ "databases": { "media": "DB_PATH" },
++ "env": {
++ "PORT": "{port}",
++ "HOST": "127.0.0.1",
++ "MEDIA_DRILL": "1",
++ "VOD_PATH": "{tmp}/storage/vods",
++ "CLIPS_PATH": "{tmp}/storage/clips",
++ "PASTES_PATH": "{tmp}/storage/pastes",
++ "ASSETS_PATH": "{tmp}/storage/assets",
++ "FILES_PATH": "{tmp}/storage/files",
++ "OBJECTS_PATH": "{tmp}/storage/objects",
++ "EVENTS_URL": "",
++ "OV_OAUTH_CLIENT_SECRET": ""
++ },
++ "requires": [
++ { "file": "server/drill.js", "contains": "MEDIA_DRILL" },
++ { "file": "server/public/routes.js", "contains": "refuseBytes" }
++ ],
++ "compare": [
++ "/browse?tab=videos",
++ "/browse?tab=clips"
++ ],
++ "counts": [
++ { "db": "media", "table": "media_objects" },
++ { "db": "media", "table": "vods" },
++ { "db": "media", "table": "clips" },
++ { "db": "media", "table": "apps" }
++ ]
}
},
"tools": {
ddiff --git a/test/drill-live-media.test.js b/test/drill-live-media.test.js
nnew file mode 100644
iindex 0000000..c5b007d
---- /dev/null
++++ b/test/drill-live-media.test.js
@@@ -0,0 +1,243 @@
++'use strict';
++/**
++ * The example inventory's live and media drills (LIVE_DRILL, OpenVibe.Live 8a58aea; MEDIA_DRILL,
++ * OpenVibe.Media 6e74eb0), run by `ovhost drill` on the fake host:
++ * - live: the production unit's ExecStart from its own checkout with LIVE_DRILL=1, the copy as
++ * DB_PATH and {tmp}/data as DATA_DIR (created before the start); DB-only reads compared, the
++ * values production changes while someone is live ignored; the socket and the service untouched.
++ * - media: MEDIA_DRILL=1, the copy as DB_PATH, every storage path but the thumbnail listing moved
++ * under {tmp}/storage and never created; the /browse pages compared byte for byte.
++ * - a checkout without the switch is refused before anything is created (an older release would
++ * ignore LIVE_DRILL / MEDIA_DRILL and run its side effects).
++ */
++const assert = require('assert');
++const fs = require('fs');
++const path = require('path');
++const { scenario, test, runTests, SECRET } = require('./helpers');
++const { normalise } = require('../lib/inventory');
++
++const EXAMPLE = JSON.parse(fs.readFileSync(path.join(__dirname, '..', 'host.example.json'), 'utf8'));
++const LIVE_DB = '/opt/openvibe.live/data/live.db';
++const MEDIA_DB = '/opt/openvibe.media/data/media.db';
++
++const envOf = (text) => Object.fromEntries(text.trim().split('\n').filter((l) => !l.startsWith('#')).map((l) => [l.slice(0, l.indexOf('=')), l.slice(l.indexOf('=') + 1)]));
++const stateChanges = (host) => host.calls.filter((c) => c.cmd === 'systemctl' && !['show', 'is-active', 'is-enabled', 'cat', 'status', 'list-units'].includes(c.args[0]));
++
++/** Production answers and the drill instance's answers for the compared paths. */
++function liveBodies({ drill }) {
++ const stream = { id: 7, title: 'Morning walk', user_id: 327, username: 'japaneseoldguy', protocol: 'webrtc', is_live: 1, channel: { id: 105, user_id: 327, title: 'Walks' } };
++ const live = drill
++ ? { ...stream, viewer_count: 0, peak_viewers: 3, external_viewer_count: 0, total_viewer_count: 0, last_heartbeat: '2026-09-24 10:00:00', thumbnail_url: 'https://openvibe.media/t/stream-7-1.jpg' }
++ : { ...stream, viewer_count: 12, peak_viewers: 14, external_viewer_count: 5, total_viewer_count: 17, last_heartbeat: '2026-09-24 10:00:30', thumbnail_url: 'https://openvibe.media/t/stream-7-2.jpg' };
++ const { external_viewer_count, total_viewer_count, channel, ...row } = live;
++ const slot = { managed_stream_id: 3, slug: 'walks', title: 'Walks', last_live_at: '2026-09-23 21:00:00', vod_thumbnail: drill ? null : 'https://openvibe.media/t/vod-99-1.jpg' };
++ return {
++ '/api/themes': '{"themes":[{"id":1,"slug":"vibe","name":"Vibe"}]}',
++ '/api/emotes/global': '{"emotes":[{"id":"custom-1","code":"ovWave"}]}',
++ '/api/streams': JSON.stringify({ streams: [live] }),
++ '/api/streams/recently-online?limit=20': JSON.stringify({ streamers: [{ user_id: 327, username: 'japaneseoldguy', managed_streams: [slot] }], total: 1, limit: 20, offset: 0, hasMore: false }),
++ // The channel's live-only answer carries the stream row, its endpoint and recording state; no external counts.
++ '/api/streams/channel/japaneseoldguy/live': JSON.stringify({ channel: { username: 'JapaneseOldGuy', display_name: 'JapaneseOldGuy', user_id: 327 }, streams: [{ ...row, server_clip: !drill, endpoint: { roomId: 'stream-7' } }] }),
++ };
++}
++
++async function liveHost({ switchInCheckout = true, drillBodies = null } = {}) {
++ const host = scenario();
++ const doc = JSON.parse(host.read('/etc/openvibe/host.json'));
++ doc.services.live = JSON.parse(JSON.stringify(EXAMPLE.services.live));
++ host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });
++ if (switchInCheckout) {
++ host.put('/opt/openvibe.live/server/drill.js', "const enabled = parse(process.env.LIVE_DRILL);\n", { owner: 'ubuntu' });
++ host.put('/opt/openvibe.live/server/paths.js', "return path.resolve(process.env.DATA_DIR || './data');\n", { owner: 'ubuntu' });
++ }
++ host.put('/etc/systemd/system/openvibe-live.service', [
++ '[Service]', 'Type=simple', 'User=ubuntu', 'WorkingDirectory=/opt/openvibe.live', 'EnvironmentFile=/etc/openvibe/live.env',
++ 'Environment=NODE_ENV=production', 'Environment=PATH=/usr/local/bin:/usr/bin:/bin',
++ 'ExecStart=/usr/bin/env node /opt/openvibe.live/server/index.js', 'ReadWritePaths=/opt/openvibe.live/data', '',
++ ].join('\n'));
++ host.put('/etc/systemd/system/openvibe-live.service.d/socket.conf', '[Unit]\nRequires=openvibe-live.socket\nAfter=openvibe-live.socket\n');
++ const unit = host.units.get('openvibe-live.service');
++ unit.dropIns = ['/etc/systemd/system/openvibe-live.service.d/socket.conf'];
++ host.alivePids.add(unit.mainPid);
++ host.listeners.set(3000, [{ pid: unit.mainPid, process: 'node' }]);
++ host.put(LIVE_DB, 'sqlite-production', { owner: 'ubuntu', mode: 0o640 });
++
++ const prod = liveBodies({ drill: false });
++ host.http.set('http://127.0.0.1:3000/api/ready', () => ({ status: 200, body: { ready: true } }));
++ for (const [p, body] of Object.entries(prod)) host.http.set(`http://127.0.0.1:3000${p}`, () => ({ status: 200, body }));
++ const counts = { users: 393, channels: 180, managed_streams: 140, streams: 5120, follows: 910, chat_messages: 250000 };
++ host.sqliteHandler = (db, sql) => {
++ if (/integrity_check/.test(sql)) return [{ integrity_check: 'ok' }];
++ const m = /FROM "([a-z_]+)"/.exec(sql);
++ return [{ n: m ? counts[m[1]] ?? 0 : 0 }];
++ };
++ host.onSystemdRun = (spec) => {
++ const envFile = spec.envFiles[spec.envFiles.length - 1];
++ host.drillEnv = host.read(envFile);
++ host.dataDirAtStart = !!host.files.get(path.join(path.dirname(envFile), 'data'));
++ host.listeners.set(13000, [{ pid: spec.pid, process: 'node' }]);
++ const alive = (fn) => () => (host.alivePids.has(spec.pid) ? fn() : { status: 0, error: 'ECONNREFUSED' });
++ host.http.set('http://127.0.0.1:13000/api/ready', alive(() => ({ status: 200, body: { ready: true, mode: 'drill' } })));
++ for (const [p, body] of Object.entries(drillBodies || liveBodies({ drill: true }))) host.http.set(`http://127.0.0.1:13000${p}`, alive(() => ({ status: 200, body })));
++ return undefined;
++ };
++ const b = await host.cli('backup', 'live', '--json');
++ assert.strictEqual(b.code, 0, b.out);
++ host.advance(60 * 1000);
++ return host;
++}
++
++const BROWSE = (tab) => `<!DOCTYPE html><html><body><nav class="tabs"><a href="/?tab=${tab}">Videos<span class="n">1,204</span></a><a>Thumbnails<span class="n">5,120</span></a></nav><div class="grid">…</div></body></html>`;
++
++async function mediaHost({ switchInCheckout = true } = {}) {
++ const host = scenario();
++ const doc = JSON.parse(host.read('/etc/openvibe/host.json'));
++ doc.services.media = JSON.parse(JSON.stringify(EXAMPLE.services.media));
++ host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });
++ if (switchInCheckout) {
++ host.put('/opt/openvibe.media/server/drill.js', "const enabled = parse(process.env.MEDIA_DRILL);\n", { owner: 'ubuntu' });
++ host.put('/opt/openvibe.media/server/public/routes.js', "if (drill.refuseBytes(res)) return;\n", { owner: 'ubuntu' });
++ }
++ host.put('/etc/systemd/system/openvibe-media.service', [
++ '[Service]', 'Type=simple', 'User=ubuntu', 'WorkingDirectory=/opt/openvibe.media', 'EnvironmentFile=/etc/openvibe/media.env',
++ 'ExecStart=/usr/bin/env node /opt/openvibe.media/server/index.js', 'ReadWritePaths=/opt/openvibe.media/data', '',
++ ].join('\n'));
++ const unit = host.units.get('openvibe-media.service');
++ host.alivePids.add(unit.mainPid);
++ host.listeners.set(4100, [{ pid: unit.mainPid, process: 'node' }]);
++ host.put(MEDIA_DB, 'sqlite-production', { owner: 'ubuntu', mode: 0o640 });
++ host.http.set('http://127.0.0.1:4100/api/ready', () => ({ status: 200, body: { ready: true } }));
++ for (const tab of ['videos', 'clips']) host.http.set(`http://127.0.0.1:4100/browse?tab=${tab}`, () => ({ status: 200, body: BROWSE(tab) }));
++ const counts = { media_objects: 6021, vods: 1300, clips: 4700, apps: 4 };
++ host.sqliteHandler = (db, sql) => {
++ if (/integrity_check/.test(sql)) return [{ integrity_check: 'ok' }];
++ if (/is_recording/.test(sql)) return [{ n: 0 }];
++ const m = /FROM "([a-z_]+)"/.exec(sql);
++ return [{ n: m ? counts[m[1]] ?? 0 : 0 }];
++ };
++ host.onSystemdRun = (spec) => {
++ const envFile = spec.envFiles[spec.envFiles.length - 1];
++ host.drillEnv = host.read(envFile);
++ host.drillDirAtStart = [...host.files.keys()].filter((f) => f.startsWith(`${path.dirname(envFile)}/`)).map((f) => path.relative(path.dirname(envFile), f)).sort();
++ host.listeners.set(14100, [{ pid: spec.pid, process: 'node' }]);
++ const alive = (fn) => () => (host.alivePids.has(spec.pid) ? fn() : { status: 0, error: 'ECONNREFUSED' });
++ host.http.set('http://127.0.0.1:14100/api/ready', alive(() => ({ status: 200, body: { ready: true, mode: 'drill' } })));
++ for (const tab of ['videos', 'clips']) host.http.set(`http://127.0.0.1:14100/browse?tab=${tab}`, alive(() => ({ status: 200, body: BROWSE(tab) })));
++ return undefined;
++ };
++ const b = await host.cli('backup', 'media', '--json');
++ assert.strictEqual(b.code, 0, b.out);
++ host.advance(60 * 1000);
++ return host;
++}
++
++runTests([
++ test('the example inventory: live and media drills are supported, on 13000 and 14100, with their switches', () => {
++ const inv = normalise(EXAMPLE);
++ const live = inv.services.live.drill;
++ assert.strictEqual(live.supported, true);
++ assert.strictEqual(live.port, 13000);
++ assert.deepStrictEqual(live.databases, { live: { env: 'DB_PATH', dir: false } });
++ assert.strictEqual(live.env.LIVE_DRILL, '1');
++ assert.strictEqual(live.env.DATA_DIR, '{tmp}/data');
++ assert.strictEqual(live.env.HOST, '127.0.0.1');
++ assert.deepStrictEqual(live.dirs, ['{tmp}/data']);
++ assert.deepStrictEqual(live.requires.map((r) => r.contains), ['LIVE_DRILL', 'DATA_DIR']);
++ assert.deepStrictEqual(live.counts.map((c) => c.table), ['users', 'channels', 'managed_streams', 'streams', 'follows', 'chat_messages']);
++ assert.ok(!live.outbound && !live.bind.length);
++ const media = inv.services.media.drill;
++ assert.strictEqual(media.supported, true);
++ assert.strictEqual(media.port, 14100);
++ assert.deepStrictEqual(media.databases, { media: { env: 'DB_PATH', dir: false } });
++ assert.strictEqual(media.env.MEDIA_DRILL, '1');
++ for (const k of ['VOD_PATH', 'CLIPS_PATH', 'PASTES_PATH', 'ASSETS_PATH', 'FILES_PATH', 'OBJECTS_PATH']) assert.match(media.env[k], /^\{tmp\}\/storage\//, k);
++ assert.ok(!('THUMBNAILS_PATH' in media.env), '/browse lists production\'s thumbnail directory');
++ assert.deepStrictEqual(media.requires.map((r) => r.contains), ['MEDIA_DRILL', 'refuseBytes']);
++ assert.deepStrictEqual(media.compare.map((c) => c.path), ['/browse?tab=videos', '/browse?tab=clips']);
++ assert.deepStrictEqual(media.counts.map((c) => c.table), ['media_objects', 'vods', 'clips', 'apps']);
++ }),
++
++ test('ovhost drill live: LIVE_DRILL=1 from the production unit, copy + DATA_DIR in the drill directory, live values ignored, production untouched', async () => {
++ const host = await liveHost();
++ const callsBefore = host.calls.length;
++ const r = await host.cli('drill', 'live');
++ assert.strictEqual(r.code, 0, r.out);
++ const rec = JSON.parse(host.read('/var/lib/openvibe-host/drills/live.jsonl').trim().split('\n').pop());
++ assert.strictEqual(rec.result, 'passed', JSON.stringify(rec.failure));
++ assert.deepStrictEqual(rec.compare.map((c) => [c.path, c.match]), [
++ ['/api/themes', true], ['/api/emotes/global', true], ['/api/streams', true],
++ ['/api/streams/recently-online?limit=20', true], ['/api/streams/channel/japaneseoldguy/live', true],
++ ]);
++ assert.deepStrictEqual(rec.databases.map((d) => d.name), ['live'], 'rs-companion.db is not restored');
++ assert.deepStrictEqual(rec.counts.map((c) => [c.table, c.match]), [['users', true], ['channels', true], ['managed_streams', true], ['streams', true], ['follows', true], ['chat_messages', true]]);
++
++ assert.strictEqual(host.systemdRuns.length, 1);
++ const run = host.systemdRuns[0];
++ assert.deepStrictEqual(run.argv, ['/usr/bin/env', 'node', '/opt/openvibe.live/server/index.js']);
++ assert.strictEqual(run.cwd, '/opt/openvibe.live');
++ assert.deepStrictEqual(run.envFiles, ['/etc/openvibe/live.env', `${rec.dir}/drill.env`]);
++ for (const p of ['SocketBindAllow=tcp:13000', 'SocketBindDeny=any', 'IPAddressDeny=any', 'IPAddressAllow=localhost', 'ProtectSystem=strict', `ReadWritePaths=${rec.dir}`, 'Environment=NODE_ENV=production']) assert.ok(run.props.includes(p), p);
++ assert.ok(!run.props.some((p) => /BindPaths=|ReadWritePaths=\/opt/.test(p)), 'no bind mounts; production\'s data directory is not writable');
++ assert.deepStrictEqual(envOf(host.drillEnv), {
++ PORT: '13000', HOST: '127.0.0.1', LIVE_DRILL: '1', DATA_DIR: `${rec.dir}/data`,
++ EVENTS_URL: '', OV_OAUTH_CLIENT_SECRET: '', MEDIA_URL: 'http://127.0.0.1:9', DB_PATH: `${rec.dir}/db/live.db`,
++ });
++ assert.ok(host.dataDirAtStart, 'DATA_DIR exists before the instance starts');
++
++ const calls = host.calls.slice(callsBefore);
++ assert.deepStrictEqual(calls.filter((c) => c.cmd === 'kill').map((c) => c.args), [['SIGTERM', run.pid]], 'only the drill instance is signalled');
++ assert.deepStrictEqual(stateChanges(host), [], 'no unit was started, stopped or restarted');
++ assert.deepStrictEqual(host.socketViolations(), []);
++ assert.strictEqual(host.units.get('openvibe-live.socket').active, 'active');
++ assert.strictEqual(host.read(LIVE_DB), 'sqlite-production');
++ assert.ok(!(r.out + JSON.stringify(host.systemdRuns) + host.read('/var/lib/openvibe-host/drills/live.jsonl')).includes(SECRET));
++ }),
++
++ test('ovhost drill live: a difference outside the ignored live values fails the drill', async () => {
++ const drill = liveBodies({ drill: true });
++ drill['/api/streams'] = drill['/api/streams'].replace('Morning walk', 'Evening walk');
++ const host = await liveHost({ drillBodies: drill });
++ const r = await host.cli('drill', 'live');
++ assert.strictEqual(r.code, 2, r.out);
++ const rec = JSON.parse(host.read('/var/lib/openvibe-host/drills/live.jsonl').trim().split('\n').pop());
++ const streams = rec.compare.find((c) => c.path === '/api/streams');
++ assert.strictEqual(streams.match, false);
++ assert.match(streams.detail, /\$\.streams\[0\]\.title/);
++ }),
++
++ test('ovhost drill media: MEDIA_DRILL=1, storage under {tmp}/storage and never created, /browse byte for byte', async () => {
++ const host = await mediaHost();
++ const r = await host.cli('drill', 'media');
++ assert.strictEqual(r.code, 0, r.out);
++ const rec = JSON.parse(host.read('/var/lib/openvibe-host/drills/media.jsonl').trim().split('\n').pop());
++ assert.strictEqual(rec.result, 'passed', JSON.stringify(rec.failure));
++ assert.deepStrictEqual(rec.compare.map((c) => [c.path, c.match, c.mode]), [['/browse?tab=videos', true, 'bytes'], ['/browse?tab=clips', true, 'bytes']]);
++ assert.deepStrictEqual(rec.counts.map((c) => [c.table, c.production, c.match]), [['media_objects', 6021, true], ['vods', 1300, true], ['clips', 4700, true], ['apps', 4, true]]);
++ const run = host.systemdRuns[0];
++ assert.deepStrictEqual(run.argv, ['/usr/bin/env', 'node', '/opt/openvibe.media/server/index.js']);
++ assert.ok(run.props.includes('SocketBindAllow=tcp:14100'));
++ const env = envOf(host.drillEnv);
++ assert.deepStrictEqual(env, {
++ PORT: '14100', HOST: '127.0.0.1', MEDIA_DRILL: '1',
++ VOD_PATH: `${rec.dir}/storage/vods`, CLIPS_PATH: `${rec.dir}/storage/clips`, PASTES_PATH: `${rec.dir}/storage/pastes`,
++ ASSETS_PATH: `${rec.dir}/storage/assets`, FILES_PATH: `${rec.dir}/storage/files`, OBJECTS_PATH: `${rec.dir}/storage/objects`,
++ EVENTS_URL: '', OV_OAUTH_CLIENT_SECRET: '', DB_PATH: `${rec.dir}/db/media.db`,
++ });
++ assert.deepStrictEqual(host.drillDirAtStart, ['db', 'db/media.db', 'drill.env'], 'nothing but the copy and the override file; storage is never created');
++ assert.deepStrictEqual(stateChanges(host), []);
++ assert.strictEqual(host.read(MEDIA_DB), 'sqlite-production');
++ assert.ok(!(r.out + JSON.stringify(host.systemdRuns)).includes(SECRET));
++ }),
++
++ test('a checkout without the drill switch is refused before anything is created', async () => {
++ for (const [id, mk, want] of [['live', liveHost, /LIVE_DRILL in server\/drill\.js \(file not found\); DATA_DIR in server\/paths\.js \(file not found\)/], ['media', mediaHost, /MEDIA_DRILL in server\/drill\.js \(file not found\)/]]) {
++ const host = await mk({ switchInCheckout: false });
++ const r = await host.cli('drill', id);
++ assert.strictEqual(r.code, 1, r.out);
++ assert.match(r.out, want);
++ assert.match(r.out, /Deploy a release with the drill switch first/);
++ assert.strictEqual(host.systemdRuns.length, 0);
++ assert.ok(![...host.files.keys()].some((f) => f.startsWith('/var/lib/openvibe-drills/')), 'no drill directory');
++ }
++ }),
++]);