openvibers/openvibe.host Commits

Restore drills for Tools and Games: one unit of a multi-unit service, data-directory databases, bind mounts for checkout-relative files, and a check that the deployed checkout has the drill switches

OP OpenVibers committed 23 Sept 2026 · 8575c912ff · 8 files, +378 −30

Host side (lib/inventory.js, lib/commands/drill.js):
- drill.unit starts one of a service's units (its ExecStart, WorkingDirectory and non-secret Environment=); drill.productionPort is the production port the comparison reads (the drilled app's, not the gateway's).
- drill.databases.<name> may be { "env": "DATA_DIR", "dir": true | "{tmp}/<name>" }: the copy keeps its production file name inside that directory and the env var points at the directory.
- drill.bind adds BindPaths= in the drill unit's own mount namespace, for files a service opens relative to its checkout with no env override (Tools yt/maps/food analytics.db); targets must be inside the checkout, sources under {tmp}.
- drill.requires refuses the drill, before anything is created, unless the deployed checkout contains each switch the overrides rely on (an older release would ignore them and run its side effects).

Inventory (host.example.json): tools drills the docs app alone with switches Tools already has (TOOLS_JOB_RESULTS=local so the job pruner cannot delete production Media objects, EVENTS_PUBLISH=off, TOOLS_JOBS_CONCURRENCY=0, uploads/output under {tmp}); games drills with EVENTS_PUBLISH=off, MEDIA_MIRROR=off and no client secret (reviewed against Games 7863fc6: one listener, tick loop flushes into the copy only). Neither has run on the host. docs/restore-drills.md lists the optional TOOLS_DRILL / GAMES_DRILL switches (with code locations) that would make both drills stricter.

Changes

ddiff --git a/README.md b/README.md
iindex 7d3218a..0ae9353 100644
---- a/README.md
++++ b/README.md
@@@ -150,7 +150,7 @@ Each inventory entry's `drill` block declares:
  - `counts`: `{ db, table }` pairs;
  - `supported: false` with a `reason` when a second instance cannot run without side effects.
  
--In `host.example.json`, 24 services have drill blocks, and 19 can be drilled. live, media, tools, games and ai are marked unsupported, and the reasons are in the file. `command` (an absolute argv) is required when a service has several units, as openre does: only `openre-api` runs in its drill.
++In `host.example.json`, 24 services have drill blocks, and 21 can be drilled. live, media and ai are marked unsupported, and the reasons are in the file. For a service with several units, `unit` (as tools does: only the docs app runs) or `command` (an absolute argv, as openre does: only `openre-api` runs) says which one process the drill starts. `productionPort` says which production port to compare against, `databases` entries can be data directories (`{ "env": "DATA_DIR", "dir": true }`), `bind` redirects paths a service opens relative to its checkout into the drill directory (`BindPaths=`, in the drill unit's own mount namespace), and `requires` refuses the drill unless the deployed checkout has each switch the overrides rely on. The Tools and Games drills rely only on switches those repositories already have. [docs/restore-drills.md](docs/restore-drills.md) lists what each repository could add to make its drill stricter.
  
  The sandbox blocks writes outside the drill directory and addresses beyond loopback. Loopback stays open, so the overrides are what keep a drill away from production services. Common overrides are `OV_OAUTH_CLIENT_SECRET=` (no service tokens) and `http://127.0.0.1:9` for URLs whose empty value would fall back to a production service.
  
ddiff --git a/docs/backups.md b/docs/backups.md
iindex d66d4db..79b2a7f 100644
---- a/docs/backups.md
++++ b/docs/backups.md
@@@ -350,7 +350,7 @@ The drill:
  
  Record the result in [restore-drills.md](restore-drills.md) and note "off-host" in the Backup column.
  
--For services without a drill (live, media, tools, games, ai), download the copy and check it by
++For services without a drill (live, media, ai), download the copy and check it by
  hand:
  
  ```
ddiff --git a/docs/restore-drills.md b/docs/restore-drills.md
iindex cffae95..931e5c0 100644
---- a/docs/restore-drills.md
++++ b/docs/restore-drills.md
@@@ -143,6 +143,78 @@ Supported with declared overrides (not yet run on the host with `ovhost drill`):
  Services that are not deployed yet (everything except network, live, media, tools, community and
  games as of 22 Sep) have entries built from their repositories' `deploy/` directories.
  
++- tools (not run yet): the **docs** app alone. Tools runs eight units; `drill.unit` picks
++  `openvibe-tools-docs.service`, whose ExecStart, WorkingDirectory and Environment= the drill copies,
++  and `drill.productionPort` (4016) is what the comparison reads, not the gateway's 4001. docs takes a
++  data DIRECTORY (`DATA_DIR`), so both restored copies land in `{tmp}/data/` under their production
++  names (`analytics.db`, `jobs.db`). Existing Tools switches make it side-effect-free:
++  `TOOLS_JOB_RESULTS=local` (the job pruner would otherwise delete production Media objects for
++  expired jobs in the copy), `EVENTS_PUBLISH=off` and an empty `EVENTS_URL` (the copy's pending outbox
++  rows would be republished), `TOOLS_JOBS_CONCURRENCY=0` (no job runs), `UPLOADS_DIR`/`OUTPUT_DIR`
++  under `{tmp}`, an empty `OV_OAUTH_CLIENT_SECRET`. Job recovery and pruning, analytics aggregation
++  and upload retention still run, on the copy only. `drill.requires` refuses the drill unless the
++  deployed checkout has those switches. Compares `/release.json`; counts `tool_jobs`.
++- games (not run yet): Reviewed against OpenVibe.Games `7863fc6`. One HTTP listener; `/ws` and
++  `/editor-ws` are upgrades on it. tsx runs `src/main.ts` directly, with no build. The unit sets
++  `DB_PATH` with `Environment=`, and that value never reaches the drill because the drill sets
++  `DB_PATH` itself. `EVENTS_PUBLISH=off` stops the outbox relay, `MEDIA_MIRROR=off` stops the
++  map-asset mirror, and an empty `OV_OAUTH_CLIENT_SECRET` stops every service call. The 30 Hz tick
++  loop keeps running because readiness needs recent ticks, and it flushes the world into the copy
++  only. `map.json` stays production's file, which is read-only in the sandbox. `drill.requires`
++  checks that `apps/server/src/config.ts` has both switches. The drill compares `/map.json` and
++  `/api/v1/mods` and counts `players` and `mods`, which the simulation does not write. There is no
++  Games backup yet, so run `sudo ovhost backup games` first.
++
++### What the drill block can declare for multi-process and checkout-relative services
++
++- `unit`: which of the service's units to start (its ExecStart, WorkingDirectory and non-secret
++  Environment=). It is required when there are several units and no `command`.
++- `productionPort`: the production port the compare paths are read from. The default is the service's
++  port.
++- `databases.<name>: { "env": "DATA_DIR", "dir": true }`, or `"dir": "{tmp}/<name>"`: the copy keeps
++  its production file name inside that directory, and the env var, if one is given, points at the
++  directory.
++- `bind: [{ "from": "{tmp}/…", "to": "<path inside the checkout>" }]`: `BindPaths=` in the drill
++  unit's own mount namespace. Use it for files a service opens relative to its checkout with no env
++  override: the restored copy appears at the production path for the drill instance only. Tools yt,
++  maps and food open `apps/<app>/data/analytics.db` this way. Drill one of them with
++  `unit: openvibe-tools-<app>.service`, a `databases` entry with `"dir": "{tmp}/app-data"`, and
++  `bind: [{ "from": "{tmp}/app-data", "to": "/opt/openvibe.tools/apps/<app>/data" }]`. For yt, also
++  set `YT_PROXY=`. Before first use, check on the host that `BindPaths=` works together with
++  `ProtectSystem=strict` in a `systemd-run` transient unit.
++- `requires: [{ "file": "<path in the checkout>", "contains": "<text>" }]`: the drill refuses before
++  it creates anything unless the deployed checkout has every switch its overrides rely on. An older
++  release would ignore the override and run its side effects.
++
++### What Tools and Games should add (product side, not Host)
++
++These make the drills stricter and quieter. Neither is needed for the drills above to be
++side-effect-free.
++
++- **Tools `TOOLS_DRILL=1`:**
++  - `_shared/jobs/system.js:576-585`: skip `recover()`, `prune()`, the prune interval and `kick()`.
++  - `_shared/jobs/index.js:83-105`: force local results and no outbox.
++  - `_shared/analytics/tracker.js:81,105-118`: no startup `DELETE` of rate tracking, no
++    flush/aggregate/prune timers, and a no-op middleware.
++  - The `retention.startCleanup()` calls (img `:273`, docs `:323`, audio `:307`), and yt's
++    `startCleanup()` and `startUpstreamProbe()` (`:280-281`, which spawns yt-dlp against YouTube).
++  - `_shared/observe.js:95-105`: no `writableDir` probe writes.
++  - A `DATA_DIR` override in yt `index.js:26`, maps `:25` and food `:20`, so those apps no longer
++    need `bind`.
++
++  With these, the restored copies stay byte-stable during a drill, and row counts other than
++  `tool_jobs` become comparable.
++- **Games `GAMES_DRILL=1`:**
++  - `config.ts:60-99`: force the client secret, OAuth, `networkAuthUrl`, `eventsUrl` and `mediaUrl`
++    to null.
++  - `config.ts:77-79`: refuse to start unless `HOST` is loopback and `DB_PATH` is not the production
++    path.
++  - `main.ts:252`: refuse `/ws` and `/editor-ws` upgrades.
++  - Return 403 for map, map-asset and mod writes (`httpServer.ts:161,231` and the `routes.ts` write
++    branches), and for `/auth/*` (`httpServer.ts:333`).
++  - Optionally skip the periodic flush (`gameServer.ts:2320,2502`) and the prune timer
++    (`main.ts:148`), so world tables can be counted too.
++
  Marked `drill: { supported: false }`, with the reason in the inventory:
  
  - **live:** `data/analytics.db` is opened relative to the checkout at boot. It is read-only in the
@@@ -151,6 +223,3 @@ Marked `drill: { supported: false }`, with the reason in the inventory:
  - **media:** webhooks to Live take their URL from the `apps` table and have no env switch. The health
    job, junk sweep and clip jobs run ffmpeg against production files through absolute paths stored in
    the database.
--- **tools:** eight units. yt, food and maps open `analytics.db` relative to the checkout. The job
--  runtimes re-queue and prune with no switch.
--- **games:** its runtime side effects have not been reviewed.
ddiff --git a/host.example.json b/host.example.json
iindex b0421aa..6f93575 100644
---- a/host.example.json
++++ b/host.example.json
@@@ -135,8 +135,35 @@
          { "name": "tools-img-jobs", "path": "/opt/openvibe.tools/apps/img/data/jobs.db" }
        ],
        "drill": {
--        "supported": false,
--        "reason": "Eight units (gateway plus seven apps) run as separate processes and a drill starts one; yt, food and maps open apps/<app>/data/analytics.db relative to the checkout with no env override (read-only in the sandbox, production's file without it); the job runtimes prune and re-queue jobs at start with no switch. Its only databases are per-app analytics."
++        "_notes": "Drills ONE app: docs (both database kinds; spawns no external program; every path has an env override). Its unit's ExecStart/WorkingDirectory come from drill.unit, and the comparison runs against docs' own port. DATA_DIR takes a directory, so both copies land in {tmp}/data under their production names. TOOLS_JOB_RESULTS=local keeps the job pruner from deleting production Media objects for expired jobs in the copy; EVENTS_PUBLISH=off keeps the copy's pending outbox rows from being republished; TOOLS_JOBS_CONCURRENCY=0 runs no job. What still runs, on the copy only: job recovery and pruning, analytics aggregation, upload retention (all under {tmp}). yt, maps and food open data/analytics.db relative to the checkout: drill one with drill.unit + a {tmp} data dir + bind over apps/<app>/data (plus YT_PROXY= for yt) until Tools adds DATA_DIR there. Not run on the host yet.",
++        "port": 14016,
++        "unit": "openvibe-tools-docs.service",
++        "productionPort": 4016,
++        "databases": {
++          "tools-docs-analytics": { "env": "DATA_DIR", "dir": true },
++          "tools-docs-jobs": { "env": "DATA_DIR", "dir": true }
++        },
++        "env": {
++          "PORT": "{port}",
++          "HOST": "127.0.0.1",
++          "UPLOADS_DIR": "{tmp}/uploads",
++          "OUTPUT_DIR": "{tmp}/output",
++          "TOOLS_JOBS_CONCURRENCY": "0",
++          "TOOLS_JOB_RESULTS": "local",
++          "EVENTS_PUBLISH": "off",
++          "EVENTS_URL": "",
++          "OV_OAUTH_CLIENT_SECRET": "",
++          "MEDIA_URL": "http://127.0.0.1:9"
++        },
++        "dirs": ["{tmp}/uploads", "{tmp}/output"],
++        "requires": [
++          { "file": "apps/docs/server/config.js", "contains": "DATA_DIR" },
++          { "file": "apps/_shared/jobs/index.js", "contains": "TOOLS_JOB_RESULTS" },
++          { "file": "apps/_shared/jobs/events.js", "contains": "EVENTS_PUBLISH" }
++        ],
++        "ready": "/api/ready",
++        "compare": ["/release.json"],
++        "counts": [{ "db": "tools-docs-jobs", "table": "tool_jobs" }]
        }
      },
      "community": {
@@@ -204,8 +231,28 @@
        "ready": { "url": "http://127.0.0.1:8000/api/ready", "timeoutSeconds": 60 },
        "databases": [{ "name": "world", "path": "/opt/openvibe.games/data/world.db" }],
        "drill": {
--        "supported": false,
--        "reason": "Not reviewed: a pnpm workspace with a TypeScript build whose runtime side effects (world simulation loop, realtime ports) have not been read, so no side-effect-free overrides can be declared without guessing."
++        "_notes": "Reviewed 2026-09-23 (OpenVibe.Games 7863fc6): one HTTP listener (PORT/HOST; /ws and /editor-ws are upgrades on it), tsx runs src/main.ts with no build, and the unit's DB_PATH (Environment=) never reaches the drill because the drill sets it. EVENTS_PUBLISH=off stops the outbox relay, MEDIA_MIRROR=off the map-asset mirror, an empty OV_OAUTH_CLIENT_SECRET every service call. The 30 Hz tick loop keeps running (readiness needs it) and flushes the world into the copy only; map.json stays production's file, read-only in the sandbox. Count only tables the simulation does not write. Not run on the host yet; take `ovhost backup games` first.",
++        "port": 18000,
++        "databases": { "world": "DB_PATH" },
++        "env": {
++          "PORT": "{port}",
++          "HOST": "127.0.0.1",
++          "OV_OAUTH_CLIENT_SECRET": "",
++          "EVENTS_PUBLISH": "off",
++          "EVENTS_URL": "",
++          "MEDIA_MIRROR": "off",
++          "TMPDIR": "{tmp}/tmp"
++        },
++        "dirs": ["{tmp}/tmp"],
++        "requires": [
++          { "file": "apps/server/src/config.ts", "contains": "EVENTS_PUBLISH" },
++          { "file": "apps/server/src/config.ts", "contains": "MEDIA_MIRROR" }
++        ],
++        "compare": ["/map.json", "/api/v1/mods"],
++        "counts": [
++          { "db": "world", "table": "players" },
++          { "db": "world", "table": "mods" }
++        ]
        }
      },
      "sites": {
ddiff --git a/lib/commands/drill.js b/lib/commands/drill.js
iindex 0018dd3..c0d4b01 100644
---- a/lib/commands/drill.js
++++ b/lib/commands/drill.js
@@@ -159,7 +159,8 @@ async function resolveBackup(exec, inv, svc, dirArg) {
          const file = path.join(dir, `${name}${path.extname(db.path) || '.db'}`);
          const fst = await exec.stat(file);
          if (!fst || !fst.isFile) throw new DrillError(`backup ${dir} has no copy of the ${name} database (${path.basename(file)})`);
--        files.push({ name, file, production: db.path, envName: svc.drill.databases[name], ext: path.extname(db.path) || '.db' });
++        const target = svc.drill.databases[name];
++        files.push({ name, file, production: db.path, envName: target.env, dir: target.dir, ext: path.extname(db.path) || '.db' });
      }
      return { dir, takenAt, files };
  }
@@@ -168,10 +169,11 @@ async function resolveBackup(exec, inv, svc, dirArg) {
  async function startSpec(exec, svc) {
      const d = svc.drill;
      if (!svc.units.length && !d.command) throw new DrillError(`${svc.id} has no unit to copy ExecStart from; set drill.command`);
--    if (svc.units.length > 1 && !d.command) throw new DrillError(`${svc.id} runs ${svc.units.length} units; a drill starts one process, so set drill.command (or mark the drill unsupported)`);
++    if (svc.units.length > 1 && !d.command && !d.unit) throw new DrillError(`${svc.id} runs ${svc.units.length} units; a drill starts one process, so set drill.unit (or drill.command, or mark the drill unsupported)`);
++    const unit = d.unit || svc.units[0];
      let directives = { execStart: null, rawExecStart: null, workingDirectory: null, user: null, environment: [] };
--    if (svc.units.length) {
--        const st = await systemd.show(exec, svc.units[0]);
++    if (unit) {
++        const st = await systemd.show(exec, unit);
          const texts = [];
          for (const f of [st.fragmentPath, ...st.dropIns].filter(Boolean)) {
              const t = await exec.readFile(f, { privileged: true });
@@@ -181,8 +183,8 @@ async function startSpec(exec, svc) {
      }
      let argv = d.command;
      if (!argv) {
--        if (!directives.execStart || !directives.execStart.length) throw new DrillError(`no ExecStart found for ${svc.units[0]}; set drill.command`);
--        if (/[$%]/.test(directives.rawExecStart) || /^[^-:+!]*@/.test(directives.rawExecStart.split(/\s/)[0])) throw new DrillError(`${svc.units[0]} ExecStart uses substitutions or specifiers; set drill.command`);
++        if (!directives.execStart || !directives.execStart.length) throw new DrillError(`no ExecStart found for ${unit}; set drill.command`);
++        if (/[$%]/.test(directives.rawExecStart) || /^[^-:+!]*@/.test(directives.rawExecStart.split(/\s/)[0])) throw new DrillError(`${unit} ExecStart uses substitutions or specifiers; set drill.command`);
          argv = directives.execStart;
      }
      if (!path.isAbsolute(argv[0])) throw new DrillError(`the drill command must start with an absolute path (got ${argv[0]}); set drill.command`);
@@@ -192,7 +194,20 @@ async function startSpec(exec, svc) {
          if (envfile.SECRET_NAME_RE.test(e.name)) skippedEnvironment.push(e.name);
          else environment.push(e);
      }
--    return { argv, cwd: directives.workingDirectory || svc.repo, unitUser: directives.user, environment, skippedEnvironment };
++    return { unit, argv, cwd: directives.workingDirectory || svc.repo, unitUser: directives.user, environment, skippedEnvironment };
++}
++
++/**
++ * The deployed checkout must have every switch the drill's overrides rely on: an older release
++ * would ignore them and run its side effects against the world. -> the missing requirements
++ */
++async function missingRequirements(exec, svc) {
++    const missing = [];
++    for (const r of svc.drill.requires) {
++        const text = await exec.readFile(path.join(svc.repo, r.file));
++        if (text == null || !text.includes(r.contains)) missing.push(`${r.contains} in ${r.file}${text == null ? ' (file not found)' : ''}`);
++    }
++    return missing;
  }
  
  // ── the drill ────────────────────────────────────────────────────────────────
@@@ -204,13 +219,15 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
      if (!svc.drill) throw new DrillError(`${id} has no drill block in the inventory`);
      if (!svc.drill.supported) throw new DrillError(`${id} does not support restore drills: ${svc.drill.reason}`);
      const d = svc.drill;
--    if (d.compare.length && !svc.port) throw new DrillError(`${id} declares compare paths but no production port`);
++    if (d.compare.length && !d.productionPort) throw new DrillError(`${id} declares compare paths but no production port`);
++    const missing = await missingRequirements(exec, svc);
++    if (missing.length) throw new DrillError(`${id}: the deployed checkout (${svc.repo}) does not have what this drill's overrides rely on: ${missing.join('; ')}. Deploy a release with the drill switch first.`);
      if (svc.envFile && !(await exec.stat(svc.envFile))) throw new DrillError(`${svc.envFile} not found`);
      const busy = await exec.listeners(d.port);
      if (busy.length) throw new DrillError(`port ${d.port} is already in use (${busy.map((l) => `${l.process || '?'} pid ${l.pid || '?'}`).join(', ')}); free it or change drill.port`);
      const bk = await resolveBackup(exec, inv, svc, backupArg);
      const spec = await startSpec(exec, svc);
--    if (spec.unitUser && spec.unitUser !== svc.runAs) log(`warning: ${svc.units[0]} runs as ${spec.unitUser}, the inventory says ${svc.runAs}; the drill runs as ${svc.runAs}`);
++    if (spec.unitUser && spec.unitUser !== svc.runAs) log(`warning: ${spec.unit} runs as ${spec.unitUser}, the inventory says ${svc.runAs}; the drill runs as ${svc.runAs}`);
      if (spec.skippedEnvironment.length) log(`not passing on unit Environment= with secret-looking names: ${spec.skippedEnvironment.join(', ')}`);
  
      const held = await lock.acquire(exec, inv, `${id}-drill`);
@@@ -249,8 +266,12 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
          created = true;
          await exec.mkdir(path.join(tmp, 'db'), { owner: svc.runAs, mode: 0o700 });
          const vars = { tmp, port: d.port, db: {} };
--        for (const f of bk.files) vars.db[f.name] = path.join(tmp, 'db', `${f.name}${f.ext}`);
++        // A data-directory database keeps its production file name: the service finds it by name.
++        const dirOf = (f) => (f.dir ? expand(f.dir, vars) : null);
++        for (const f of bk.files) vars.db[f.name] = f.dir ? path.join(dirOf(f), path.basename(f.production)) : path.join(tmp, 'db', `${f.name}${f.ext}`);
++        for (const dir of new Set(bk.files.filter((f) => f.dir).map(dirOf))) await exec.mkdir(dir, { owner: svc.runAs, mode: 0o700 });
          for (const dir of d.dirs) await exec.mkdir(expand(dir, vars), { owner: svc.runAs, mode: 0o700 });
++        for (const b of d.bind) if (!(await exec.stat(expand(b.from, vars)))) await exec.mkdir(expand(b.from, vars), { owner: svc.runAs, mode: 0o700 });
  
          // 1. restore and check every database copy
          for (const f of bk.files) {
@@@ -275,13 +296,15 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
          // 2. start the drill instance
          const lines = ['# written by ovhost drill; no secret values: the production env file is loaded separately'];
          for (const [k, v] of Object.entries(d.env)) lines.push(envLine(k, expand(v, vars)));
--        for (const f of bk.files) lines.push(envLine(f.envName, vars.db[f.name]));
++        const dbEnv = new Map();
++        for (const f of bk.files) if (f.envName) dbEnv.set(f.envName, f.dir ? dirOf(f) : vars.db[f.name]);
++        for (const [k, v] of dbEnv) lines.push(envLine(k, v));
          const envPath = path.join(tmp, 'drill.env');
          await exec.writeFile(envPath, `${lines.join('\n')}\n`, { mode: 0o640 });
          // A unit Environment= the drill sets itself (its port, a restored database path) is not passed
          // on at all: the override file would win anyway (EnvironmentFile= beats Environment=), but a
          // production path must never reach the drill instance's environment.
--        const drillNames = new Set([...Object.keys(d.env), ...bk.files.map((f) => f.envName)]);
++        const drillNames = new Set([...Object.keys(d.env), ...dbEnv.keys()]);
          const props = [
              'Type=exec',
              ...spec.environment.filter((e) => !drillNames.has(e.name)).map((e) => environmentProperty(e.name, e.value)),
@@@ -294,6 +317,9 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
              'PrivateTmp=yes',
              'ProtectHome=read-only',
              'NoNewPrivileges=yes',
++            // Paths the service opens relative to its checkout, redirected into the drill directory
++            // inside this unit's own mount namespace; production's files are never opened.
++            ...d.bind.map((b) => `BindPaths=${expand(b.from, vars)}:${b.to}`),
              `SocketBindAllow=tcp:${d.port}`,
              'SocketBindDeny=any',
              `RuntimeMaxSec=${d.runtimeMaxSeconds}`,
@@@ -305,6 +331,7 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
          for (const p of props) args.push('-p', p);
          args.push('--', ...spec.argv.map((a) => expand(a, vars)));
          for (const u of svc.units) { const st = await systemd.show(exec, u); if (st.mainPid) productionPids.push(st.mainPid); }
++        if (d.bind.length) log(`bind mounts in the drill instance only: ${d.bind.map((b) => `${expand(b.from, vars)} → ${b.to}`).join(', ')}`);
          log(`starting ${spec.argv.join(' ')} in ${spec.cwd} as ${svc.runAs} on port ${d.port} (${unitName})`);
          const run = await exec.run('systemd-run', args, { privileged: true });
          if (run.code !== 0) throw fail('start', `systemd-run failed: ${(run.stderr || '').trim().slice(0, 300)}`);
@@@ -335,7 +362,7 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
  
          for (const c of d.compare) {
              const h = { ...headers, ...c.headers };
--            const prod = await exec.http(`http://127.0.0.1:${svc.port}${c.path}`, { timeoutMs: 10000, headers: h });
++            const prod = await exec.http(`http://127.0.0.1:${d.productionPort}${c.path}`, { timeoutMs: 10000, headers: h });
              const drl = await exec.http(`http://127.0.0.1:${d.port}${c.path}`, { timeoutMs: 10000, headers: h });
              const res = compareBodies(c, prod, drl);
              record.compare.push(res);
ddiff --git a/lib/inventory.js b/lib/inventory.js
iindex 8bec135..6964f16 100644
---- a/lib/inventory.js
++++ b/lib/inventory.js
@@@ -143,11 +143,25 @@ const PLACEHOLDER_RE = /\{(tmp|port|db:[A-Za-z0-9._-]+)\}/g;
   *
   *   supported   false (with a reason) when a second instance cannot run safely
   *   port        spare loopback port for the drill instance (never the service's own port)
-- *   databases   { <database name>: <ENV VAR that points the service at that file> }
++ *   databases   { <database name>: <ENV VAR that points the service at that file> }, or
++ *               { <database name>: { "env": "DATA_DIR", "dir": true } } when the service takes a data
++ *               DIRECTORY: the copy lands in {tmp}/data/ (or "dir": "{tmp}/<name>") under its
++ *               production file name and the env var (optional) points at that directory. Without
++ *               "env", pair the directory with `bind`.
   *   env         { NAME: value } overrides that turn side effects off; values may use {tmp}, {port}
   *               and {db:<name>}
   *   dirs        directories to create (owned by the service user) inside {tmp} before the start
   *   command     argv to start the service; default: the production unit's ExecStart
++ *   unit        which of the service's units to start (its ExecStart, WorkingDirectory and
++ *               Environment=); required with several units unless `command` is set
++ *   productionPort  the production port the compare paths are fetched from (default: the service's
++ *               port); set it when `unit` is not the one listening on that port
++ *   bind        [{ from: "{tmp}/…", to: "<absolute path inside the checkout>" }] bind mounts in the
++ *               drill instance's own mount namespace, for paths the service opens relative to its
++ *               checkout with no env override (production never sees them)
++ *   requires    [{ file: "<path in the checkout>", contains: "<text>" }] the drill refuses to start
++ *               unless the deployed checkout has each switch it relies on (an older release would
++ *               ignore the override and run its side effects)
   *   ready       readiness path; default: the path of the service's ready URL
   *   compare     [{ path, ignore: [volatile keys], headers }] read-only GETs compared with production
   *   counts      [{ db, table }] row counts compared between production and the restored copy
@@@ -165,26 +179,71 @@ function normaliseDrill(raw, svc, where) {
      if (port === svc.port) throw new InventoryError(`${w}.port must differ from the service's own port`);
      if (!svc.databases.length) throw new InventoryError(`${w}: a drill restores databases, and ${svc.id} declares none`);
      const dbNames = svc.databases.map((d) => d.name);
--    const databases = raw.databases || {};
--    if (typeof databases !== 'object' || Array.isArray(databases) || !Object.keys(databases).length) throw new InventoryError(`${w}.databases must map at least one database name to an env var name`);
--    for (const [name, envName] of Object.entries(databases)) {
++    const rawDatabases = raw.databases || {};
++    if (typeof rawDatabases !== 'object' || Array.isArray(rawDatabases) || !Object.keys(rawDatabases).length) throw new InventoryError(`${w}.databases must map at least one database name to an env var name`);
++    const databases = {};
++    const inDataDir = new Map();
++    for (const [name, target] of Object.entries(rawDatabases)) {
          if (!dbNames.includes(name)) throw new InventoryError(`${w}.databases: "${name}" is not one of ${svc.id}'s databases (${dbNames.join(', ')})`);
--        if (!ENV_NAME_RE.test(envName)) throw new InventoryError(`${w}.databases.${name} must be an env var name`);
++        if (typeof target === 'string') {
++            if (!ENV_NAME_RE.test(target)) throw new InventoryError(`${w}.databases.${name} must be an env var name`);
++            databases[name] = { env: target, dir: false };
++            continue;
++        }
++        const dir = target && target.dir === true ? '{tmp}/data' : target && target.dir;
++        if (!target || typeof target !== 'object' || typeof dir !== 'string') throw new InventoryError(`${w}.databases.${name} must be an env var name or { "env": "<NAME>", "dir": true }`);
++        if (!/^\{tmp\}\/[A-Za-z0-9._-]+$/.test(dir)) throw new InventoryError(`${w}.databases.${name}.dir must be true or "{tmp}/<name>"`);
++        if (target.env != null && !ENV_NAME_RE.test(target.env)) throw new InventoryError(`${w}.databases.${name}.env must be an env var name`);
++        const dest = `${dir}/${path.basename(svc.databases.find((d) => d.name === name).path)}`;
++        if (inDataDir.has(dest)) throw new InventoryError(`${w}.databases: ${name} and ${inDataDir.get(dest)} would both be ${dest}`);
++        inDataDir.set(dest, name);
++        databases[name] = { env: target.env || null, dir };
++    }
++    const dirEnvs = new Map();
++    for (const t of Object.values(databases)) {
++        if (!t.env) continue;
++        const key = t.dir || `file:${t.env}`;
++        if (dirEnvs.has(t.env) && dirEnvs.get(t.env) !== key) throw new InventoryError(`${w}.databases: ${t.env} would point at two different places`);
++        dirEnvs.set(t.env, key);
      }
      const env = raw.env || {};
      for (const [k, v] of Object.entries(env)) {
          if (!ENV_NAME_RE.test(k)) throw new InventoryError(`${w}.env: "${k}" is not an env var name`);
          if (typeof v !== 'string' || /[\n\r\0]/.test(v)) throw new InventoryError(`${w}.env.${k} must be a one-line string`);
          for (const m of v.matchAll(PLACEHOLDER_RE)) {
--            if (m[1].startsWith('db:') && !Object.keys(databases).includes(m[1].slice(3))) throw new InventoryError(`${w}.env.${k}: {${m[1]}} is not a restored database`);
++            if (m[1].startsWith('db:') && !Object.keys(rawDatabases).includes(m[1].slice(3))) throw new InventoryError(`${w}.env.${k}: {${m[1]}} is not a restored database`);
          }
      }
      if (!Object.values(env).some((v) => v.includes('{port}')) && !(raw.command || []).some((a) => String(a).includes('{port}'))) throw new InventoryError(`${w}.env must point the instance at its port with {port} (e.g. "PORT": "{port}")`);
--    for (const envName of Object.values(databases)) if (envName in env) throw new InventoryError(`${w}.env.${envName} is already set by drill.databases`);
++    for (const t of Object.values(databases)) if (t.env && t.env in env) throw new InventoryError(`${w}.env.${t.env} is already set by drill.databases`);
      const dirs = raw.dirs || [];
      for (const d of dirs) if (typeof d !== 'string' || !/^\{tmp\}\/[A-Za-z0-9._/-]+$/.test(d) || d.includes('..')) throw new InventoryError(`${w}.dirs entries must be paths under {tmp}`);
      if (raw.command != null && (!Array.isArray(raw.command) || !raw.command.length || raw.command.some((a) => typeof a !== 'string'))) throw new InventoryError(`${w}.command must be an argv array`);
      if (raw.command && !path.isAbsolute(raw.command[0])) throw new InventoryError(`${w}.command[0] must be an absolute path`);
++    let unit = null;
++    if (raw.unit != null) {
++        if (!svc.units.includes(raw.unit)) throw new InventoryError(`${w}.unit must be one of ${svc.id}'s units (${svc.units.join(', ') || 'none'})`);
++        unit = raw.unit;
++    }
++    let productionPort = svc.port || null;
++    if (raw.productionPort != null) {
++        productionPort = Number(raw.productionPort);
++        if (!(Number.isInteger(productionPort) && productionPort >= 1 && productionPort < 65536)) throw new InventoryError(`${w}.productionPort must be a TCP port`);
++        if (productionPort === port) throw new InventoryError(`${w}.productionPort must differ from the drill port`);
++    }
++    const bind = (raw.bind || []).map((b, i) => {
++        const from = b && b.from;
++        const to = b && b.to;
++        if (typeof from !== 'string' || !/^\{tmp\}\/[A-Za-z0-9._/-]+$/.test(from) || from.includes('..')) throw new InventoryError(`${w}.bind[${i}].from must be a path under {tmp}`);
++        if (typeof to !== 'string' || !path.isAbsolute(to) || to.includes('..') || !/^\/[A-Za-z0-9._/-]+$/.test(to)) throw new InventoryError(`${w}.bind[${i}].to must be a plain absolute path`);
++        if (!svc.repo || !(to === svc.repo || to.startsWith(`${svc.repo}/`))) throw new InventoryError(`${w}.bind[${i}].to must be inside the checkout (${svc.repo}): a drill only redirects paths the service opens relative to it`);
++        return { from, to };
++    });
++    const requires = (raw.requires || []).map((r, i) => {
++        if (!r || typeof r.file !== 'string' || path.isAbsolute(r.file) || r.file.split('/').includes('..') || !/^[A-Za-z0-9._/-]+$/.test(r.file)) throw new InventoryError(`${w}.requires[${i}].file must be a path inside the checkout`);
++        if (typeof r.contains !== 'string' || !r.contains.trim()) throw new InventoryError(`${w}.requires[${i}].contains must be the text the file must contain`);
++        return { file: r.file, contains: r.contains };
++    });
      let ready = raw.ready || null;
      if (!ready && svc.ready) ready = new URL(svc.ready.url).pathname;
      if (!ready || !/^\/[^/]/.test(ready)) throw new InventoryError(`${w}.ready must be a path such as /api/ready (or declare the service's ready URL)`);
@@@ -209,6 +268,10 @@ function normaliseDrill(raw, svc, where) {
          env,
          dirs,
          command: raw.command || null,
++        unit,
++        productionPort,
++        bind,
++        requires,
          ready,
          readyTimeoutSeconds,
          compare,
ddiff --git a/test/drill.test.js b/test/drill.test.js
iindex a5da90a..69382fa 100644
---- a/test/drill.test.js
++++ b/test/drill.test.js
@@@ -109,6 +109,83 @@ async function drillScenario({ drill: drillOverrides, backup = true } = {}) {
      return host;
  }
  
++/** A Tools-like service: a gateway and apps, each app its own unit and port, data directories. */
++function appsEntry(drill = {}) {
++    const d = {
++        port: 14016,
++        unit: 'openvibe-apps-docs.service',
++        productionPort: 4016,
++        databases: { 'docs-analytics': { env: 'DATA_DIR', dir: true }, 'docs-jobs': { env: 'DATA_DIR', dir: true }, 'yt-analytics': { dir: '{tmp}/yt-data' } },
++        env: { PORT: '{port}', HOST: '127.0.0.1', TOOLS_JOB_RESULTS: 'local', EVENTS_PUBLISH: 'off' },
++        bind: [{ from: '{tmp}/yt-data', to: '/opt/openvibe.apps/apps/yt/data' }],
++        requires: [{ file: 'apps/_shared/jobs/index.js', contains: 'TOOLS_JOB_RESULTS' }],
++        ready: '/api/ready',
++        compare: ['/release.json'],
++        counts: [{ db: 'docs-jobs', table: 'tool_jobs' }],
++        ...drill,
++    };
++    for (const [k, v] of Object.entries(d)) if (v === null) delete d[k];
++    return {
++        owner: 'ubuntu',
++        repo: '/opt/openvibe.apps',
++        units: ['openvibe-apps.service', 'openvibe-apps-docs.service', 'openvibe-apps-yt.service'],
++        envFile: '/etc/openvibe/apps.env',
++        port: 4001,
++        ready: { url: 'http://127.0.0.1:4001/api/ready', timeoutSeconds: 30 },
++        databases: [
++            { name: 'docs-analytics', path: '/opt/openvibe.apps/apps/docs/data/analytics.db' },
++            { name: 'docs-jobs', path: '/opt/openvibe.apps/apps/docs/data/jobs.db' },
++            { name: 'yt-analytics', path: '/opt/openvibe.apps/apps/yt/data/analytics.db' },
++        ],
++        drill: d,
++    };
++}
++
++async function appsScenario({ switchInCheckout = true, drill } = {}) {
++    const host = scenario();
++    const doc = JSON.parse(host.read('/etc/openvibe/host.json'));
++    doc.services.apps = appsEntry(drill);
++    host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });
++    const repo = host.createRepo('/opt/openvibe.apps', { owner: 'ubuntu' });
++    const sha = repo.commit({
++        'package.json': '{"name":"apps"}',
++        'apps/_shared/jobs/index.js': switchInCheckout ? "const want = env.TOOLS_JOB_RESULTS || 'local';" : 'jobs();',
++    }, { message: 'initial' });
++    repo.publish(sha);
++    repo.checkout(sha);
++    const unit = (name, dir, port) => {
++        host.addUnit(name, { mainPid: 3000 + port });
++        host.alivePids.add(3000 + port);
++        host.listeners.set(port, [{ pid: 3000 + port, process: 'node' }]);
++        host.put(`/etc/systemd/system/${name}`, ['[Service]', 'User=ubuntu', `WorkingDirectory=/opt/openvibe.apps/apps/${dir}`, 'Environment=NODE_ENV=production', `Environment=PORT=${port}`, `ExecStart=/usr/bin/env node /opt/openvibe.apps/apps/${dir}/server/index.js`, ''].join('\n'));
++    };
++    unit('openvibe-apps.service', 'gateway', 4001);
++    unit('openvibe-apps-docs.service', 'docs', 4016);
++    unit('openvibe-apps-yt.service', 'yt', 4013);
++    host.put('/etc/openvibe/apps.env', `OV_OAUTH_CLIENT_SECRET=${SECRET}\n`, { mode: 0o600 });
++    for (const db of appsEntry().databases) host.put(db.path, `sqlite-${db.name}`, { owner: 'ubuntu', mode: 0o640 });
++    host.http.set('http://127.0.0.1:4001/release.json', () => ({ status: 200, body: '{"service":"gateway"}' }));
++    host.http.set('http://127.0.0.1:4016/release.json', () => ({ status: 200, body: '{"service":"docs"}' }));
++    host.sqliteHandler = (db, sql) => {
++        if (/integrity_check/.test(sql)) return [{ integrity_check: 'ok' }];
++        if (/count\(\*\)/.test(sql)) return [{ n: 3 }];
++        return [{ n: 0 }];
++    };
++    host.onSystemdRun = (spec) => {
++        const envFile = spec.envFiles[spec.envFiles.length - 1];
++        host.drillEnv = host.read(envFile);
++        host.listeners.set(14016, [{ pid: spec.pid, process: 'node' }]);
++        const alive = (fn) => () => (host.alivePids.has(spec.pid) ? fn() : { status: 0, error: 'ECONNREFUSED' });
++        host.http.set('http://127.0.0.1:14016/api/ready', alive(() => ({ status: 200, body: { status: 'ready' } })));
++        host.http.set('http://127.0.0.1:14016/release.json', alive(() => ({ status: 200, body: '{"service":"docs"}' })));
++        return undefined;
++    };
++    const b = await host.cli('backup', 'apps', '--json');
++    assert.strictEqual(b.code, 0, b.out);
++    host.advance(60 * 1000);
++    return host;
++}
++
  /** Every file entry as "type:owner:mode:content", to diff the fake filesystem. */
  function fsSnapshot(host) {
      const m = new Map();
@@@ -384,6 +461,71 @@ runTests([
          assert.strictEqual(r.code, 1);
      }),
  
++    test('one app of a multi-unit service: drill.unit, data-directory databases, a bind mount, its own production port', async () => {
++        const host = await appsScenario();
++        const r = await host.cli('drill', 'apps');
++        assert.strictEqual(r.code, 0, r.out);
++        const rec = JSON.parse(host.read('/var/lib/openvibe-host/drills/apps.jsonl').trim().split('\n').pop());
++        assert.strictEqual(rec.result, 'passed', JSON.stringify(rec.failure));
++        const tmp = rec.dir;
++        // Data-directory databases keep their production file names under {tmp}/data.
++        assert.deepStrictEqual(rec.databases.map((d) => [d.name, d.copy]).sort(), [['docs-analytics', `${tmp}/data/analytics.db`], ['docs-jobs', `${tmp}/data/jobs.db`], ['yt-analytics', `${tmp}/yt-data/analytics.db`]].sort());
++        const run = host.systemdRuns[0];
++        // The docs unit's ExecStart and WorkingDirectory, not the gateway's.
++        assert.strictEqual(run.cwd, '/opt/openvibe.apps/apps/docs');
++        assert.deepStrictEqual(run.argv, ['/usr/bin/env', 'node', '/opt/openvibe.apps/apps/docs/server/index.js']);
++        assert.ok(run.props.includes('Environment=NODE_ENV=production'));
++        assert.ok(!run.props.some((p) => p.startsWith('Environment=PORT=')), 'the unit\'s PORT never reaches the drill');
++        assert.ok(run.props.includes(`BindPaths=${tmp}/yt-data:/opt/openvibe.apps/apps/yt/data`), run.props.join('\n'));
++        const env = host.drillEnv;
++        assert.match(env, new RegExp(`^DATA_DIR=${tmp}/data$`, 'm'));
++        assert.match(env, /^PORT=14016$/m);
++        assert.ok(!/analytics\.db/.test(env), 'a data-directory database is found by its file name, not an env var');
++        // Compared with the docs app's production port, not the gateway's.
++        assert.deepStrictEqual(rec.compare.map((c) => [c.path, c.match]), [['/release.json', true]]);
++        assert.ok(host.calls.some((c) => c.cmd === 'curl' && c.args[0] === 'http://127.0.0.1:4016/release.json'));
++        assert.ok(!host.calls.some((c) => c.cmd === 'curl' && c.args[0].startsWith('http://127.0.0.1:4001/')));
++        assert.ok(!host.restarts().length, 'nothing restarted');
++    }),
++
++    test('requires: a checkout without the drill switch is refused before anything is created', async () => {
++        const host = await appsScenario({ switchInCheckout: false });
++        const before = fsSnapshot(host);
++        const r = await host.cli('drill', 'apps');
++        assert.strictEqual(r.code, 1, r.out);
++        assert.match(r.out, /TOOLS_JOB_RESULTS in apps\/_shared\/jobs\/index\.js/);
++        assert.match(r.out, /Deploy a release with the drill switch first/);
++        assert.strictEqual(host.systemdRuns.length, 0);
++        assert.deepStrictEqual(changedPaths(before, fsSnapshot(host)), []);
++    }),
++
++    test('inventory: unit, productionPort, bind, requires and data-directory databases are validated', () => {
++        const entry = appsEntry();
++        const inv = (drill) => normalise({ services: { apps: { ...entry, drill: { ...entry.drill, ...drill } } } });
++        assert.strictEqual(inv({}).services.apps.drill.unit, 'openvibe-apps-docs.service');
++        assert.throws(() => inv({ unit: 'openvibe-live.service' }), /unit must be one of apps's units/);
++        assert.throws(() => inv({ productionPort: 14016 }), /productionPort must differ/);
++        assert.throws(() => inv({ bind: [{ from: '/tmp/x', to: '/opt/openvibe.apps/apps/yt/data' }] }), /bind\[0\]\.from/);
++        assert.throws(() => inv({ bind: [{ from: '{tmp}/x', to: '/etc/openvibe' }] }), /inside the checkout/);
++        assert.throws(() => inv({ bind: [{ from: '{tmp}/x', to: '/opt/openvibe.apps/../etc' }] }), /plain absolute path/);
++        assert.throws(() => inv({ requires: [{ file: '../etc/passwd', contains: 'x' }] }), /inside the checkout/);
++        assert.throws(() => inv({ requires: [{ file: 'a.js' }] }), /contains/);
++        assert.throws(() => inv({ databases: { 'docs-analytics': { env: 'DATA_DIR' } } }), /"dir": true/);
++        assert.throws(() => inv({ databases: { 'docs-analytics': { env: 'DATA_DIR', dir: true }, 'yt-analytics': { dir: true } } }), /would both be \{tmp\}\/data\/analytics\.db/);
++        assert.throws(() => inv({ databases: { 'docs-analytics': { env: 'DATA_DIR', dir: '/var/lib/x' } } }), /dir must be true or/);
++        assert.throws(() => inv({ databases: { 'docs-analytics': { env: 'DATA_DIR', dir: true }, 'docs-jobs': { env: 'DATA_DIR', dir: '{tmp}/other' } } }), /two different places/);
++        const multi = { ...entry, drill: { ...entry.drill } };
++        delete multi.drill.unit;
++        normalise({ services: { apps: multi } });   // valid inventory; the drill itself refuses (below)
++    }),
++
++    test('several units and neither drill.unit nor drill.command: refused', async () => {
++        const host = await appsScenario({ drill: { unit: null } });
++        const r = await host.cli('drill', 'apps');
++        assert.strictEqual(r.code, 1, r.out);
++        assert.match(r.out, /set drill\.unit/);
++    }),
++
      test('inventory: drill blocks are validated', () => {
          const base = { owner: 'ubuntu' };
          const svc = (drill, extra = {}) => normalise({ services: { community: { ...base, ...communityEntry(), ...extra, drill: { ...communityEntry().drill, ...drill } } } });
ddiff --git a/test/openre-workers.test.js b/test/openre-workers.test.js
iindex c2647f7..7445da6 100644
---- a/test/openre-workers.test.js
++++ b/test/openre-workers.test.js
@@@ -92,7 +92,7 @@ runTests([
          assert.strictEqual(o.drill.env.OPENRE_DRILL, '1');
          for (const k of ['EVENTS_URL', 'OV_OAUTH_CLIENT_SECRET', 'MEDIA_API_KEY']) assert.strictEqual(o.drill.env[k], '', k);
          assert.strictEqual(o.drill.env.MEDIA_URL, 'http://127.0.0.1:9');
--        assert.deepStrictEqual(o.drill.databases, { openre: 'OPENRE_DB_PATH' });
++        assert.deepStrictEqual(o.drill.databases, { openre: { env: 'OPENRE_DB_PATH', dir: false } });
          // every other service keeps an empty list
          assert.deepStrictEqual(inv.services.live.workerUnits, []);
      }),