ovhost's Media probe waits for the uploads and the open recordings too (#296)

ddiff --git a/README.md b/README.md
iindex 6b10706..009ec47 100644
---- a/README.md
++++ b/README.md
@@@ -81,7 +81,7 @@ Repository deploy scripts call ovhost. Stage A applies the deploy rules for ever
  |---|---|---|
  | Never stop `openvibe-live.socket`; restart only the service unit | Live (socket activation, `deploy/systemd`) | Every `systemctl` call goes through a guard that refuses `stop`/`restart`/`reload` on a `.socket` unit. A socket that is down gets `start`ed. The one exception is the rebind Live's script also made (`systemd.rebindSocket`, release-layout only): when the socket unit file changed or pid 1 does not hold the port, the service is stopped, the socket restarted and the service started on it. |
  | Don't restart Live while anyone is live; `--wait-idle` holds the restart | Live `deploy.sh --wait-idle` (`/api/streams`) | Protected-session probe (`http-json-count`). Checked before the checkout moves and again right before the restart. |
--| Don't restart Media while it is recording | Media hazard H3 (`vods-orphans/` in B2) | Protected-session probe (`http-json-count` of Media's `/api/ready`: `recordings_in_progress` plus `chunked_recordings_in_progress` for a browser MediaRecorder session that received a chunk in the last five minutes). An older Media without the second field is counted by the first. |
++| Don't restart Media while it is recording or uploading | Media hazard H3 (`vods-orphans/` in B2) | Protected-session probe (`http-json-count` of Media's `/api/ready`): `recordings_in_progress` (ffmpeg), `chunked_recordings_in_progress` (a browser MediaRecorder session that received a chunk in the last five minutes), `recordings_untracked_in_progress` (a recording the database still holds whose file is still growing, e.g. a chunked upload that resumed after a restart) and `uploads_in_progress` (an object upload whose bytes are moving). The extra fields are summed into the count; an older Media without one is counted by the others, and a field the service reports but cannot answer (`null`) is unknown, so the restart is held. |
  | Poll `/api/ready` after a restart; roll back if it doesn't come up | Live `deploy.sh` | Readiness poll with a timeout; on failure the checkout goes back to the previous sha, deps are reinstalled if they differed, the unit is restarted again. |
  | Install only when the lockfile or dependencies changed; restore `package-lock.json` afterwards | Network `deploy.sh` and host practice | `npm install --omit=dev` as the checkout owner, then `git checkout -- package-lock.json` if npm rewrote it. |
  | Every dependency must resolve before anything restarts | Tools `deploy.sh` (an emptied `file:` link crash-looped every unit) | `node_modules/<dep>/package.json` must exist, parse and name the package. A broken dependency is reinstalled once, and if it still fails the deploy aborts and the checkout is restored. |
@@@ -282,7 +282,7 @@ The sandbox blocks writes outside the drill directory and addresses beyond loopb
  
  ### Inventory
  
--`host.example.json` describes every service on the host. It records the checkout, owner/`runAs`, units and socket unit, `unitSources` (repo unit files), env file, `env.required` (names, or `"from-example"`), port, loopback `ready` URL (plus headers, since Tools needs `Host: openvibe.tools`), `packages` (`"apps/*"` for Tools), install command, `build` hooks, `noRestartPaths`, the `protected` probe (`http-json-count` — a numeric `field`, plus any present `addFields` —, `postgresql-count`, or `sum` of several: Live streams, Media recordings and chunk sessions, Events SSE connections, Chat sockets, Games players online, OpenRestream ingest sessions, Tools running jobs), the `drain` policy, `databases` (each a PostgreSQL database `{ "name", "engine": "postgresql", "database": "ov_<name>" }`; [docs/db-inventory.md](docs/db-inventory.md)), `backupOnChange`, `nginx`, `layout` (`release` for a `releases/<id>` + `current` service using the `release-layout` strategy) and `workerUnits`. `workerUnits` are units, or templates such as OpenRestream's `[email protected]`, that belong to the service but that ovhost **never starts, stops or restarts**: they drain and exit on their own. `status` and `validate` list their instances read-only, `validate` warns when none runs, and the inventory refuses a worker unit, or an instance of one, in `units`. The real file is `/etc/openvibe/host.json` and is **never committed**. When `ovhost` runs as root it refuses an inventory that is not root-owned or that anyone else can write, and it rejects relative paths, non-loopback probe URLs, non-SELECT probe queries and vhost names that contain a path.
++`host.example.json` describes every service on the host. It records the checkout, owner/`runAs`, units and socket unit, `unitSources` (repo unit files), env file, `env.required` (names, or `"from-example"`), port, loopback `ready` URL (plus headers, since Tools needs `Host: openvibe.tools`), `packages` (`"apps/*"` for Tools), install command, `build` hooks, `noRestartPaths`, the `protected` probe (`http-json-count` — a numeric `field`, plus any present `addFields` —, `postgresql-count`, or `sum` of several: Live streams, Media recordings, chunk sessions and uploads, Events SSE connections, Chat sockets, Games players online, OpenRestream ingest sessions, Tools running jobs), the `drain` policy, `databases` (each a PostgreSQL database `{ "name", "engine": "postgresql", "database": "ov_<name>" }`; [docs/db-inventory.md](docs/db-inventory.md)), `backupOnChange`, `nginx`, `layout` (`release` for a `releases/<id>` + `current` service using the `release-layout` strategy) and `workerUnits`. `workerUnits` are units, or templates such as OpenRestream's `[email protected]`, that belong to the service but that ovhost **never starts, stops or restarts**: they drain and exit on their own. `status` and `validate` list their instances read-only, `validate` warns when none runs, and the inventory refuses a worker unit, or an instance of one, in `units`. An `addFields` entry that the service does not report at all is read as zero (an older service), while one it reports as `null` is read as unknown: the service could not answer it, so the restart is held. The real file is `/etc/openvibe/host.json` and is **never committed**. When `ovhost` runs as root it refuses an inventory that is not root-owned or that anyone else can write, and it rejects relative paths, non-loopback probe URLs, non-SELECT probe queries and vhost names that contain a path.
  
  **Lifecycle** (roadmap WS-P task 1, `lib/lifecycle.js`). Each service's lifecycle is the `lifecycle` block of its manifest in openvibe-contracts (≥ 0.55.0): liveness, shutdown (signal, `deadlineSeconds`, drains, workers), startupRecovery, rollback, contracts and leases. A service entry may carry its own `lifecycle` block, which replaces the manifest's on this host, and `validate --manifest <file>` reads another manifest file. `ovhost validate` fails when no block is found or a field is missing, naming the service and the field (`live: lifecycle.shutdown.deadlineSeconds is missing`). It also fails when `deadlineSeconds` exceeds a unit's stop timeout (systemd's effective `TimeoutStopUSec`, else `TimeoutStopSec` in the unit source) or the unit's `KillSignal` is another signal, and when the checkout's installed openvibe-contracts is outside `contracts.range`. Worker units that drain longer than their stop timeout are a warning, since ovhost never stops them. This repository still pins openvibe-contracts v0.49.0, which has no lifecycle blocks, so until that pin moves to v0.55.0 or later `validate` reports every service as undeclared.
  
ddiff --git a/host.example.json b/host.example.json
iindex 82f8443..51e7abd 100644
---- a/host.example.json
++++ b/host.example.json
@@@ -339,9 +339,11 @@
          "url": "http://127.0.0.1:4100/api/ready",
          "field": "recordings_in_progress",
          "addFields": [
--          "chunked_recordings_in_progress"
++          "chunked_recordings_in_progress",
++          "recordings_untracked_in_progress",
++          "uploads_in_progress"
          ],
--        "label": "recordings in progress"
++        "label": "recordings and uploads in progress"
        },
        "drain": {
          "policy": "refuse",
ddiff --git a/lib/probes.js b/lib/probes.js
iindex 904627b..2c0c2bb 100644
---- a/lib/probes.js
++++ b/lib/probes.js
@@@ -5,9 +5,11 @@
   *   http-json-count  GET a loopback URL and count: `array` (dotted path) filtered by `where`
   *                    (key: value equality), or a numeric `field` (dotted path) plus every numeric
   *                    `addFields` entry that is present. An absent addField is zero (an older service
-- *                    that does not report it yet); a present one that is not a number is unknown.
++ *                    that does not report it yet); a null one is unknown (the service reports it but
++ *                    cannot answer it); a present one that is not a number is unknown.
   *                    Live: /api/streams, streams where is_live. Events: /api/ready realtime_connections.
-- *                    Media: /api/ready recordings_in_progress + chunked_recordings_in_progress.
++ *                    Media: /api/ready recordings_in_progress + chunked_recordings_in_progress +
++ *                    recordings_untracked_in_progress + uploads_in_progress.
   *   postgresql-count a read-only SELECT against a database in the cluster, run as the postgres OS
   *                    user over peer auth (no secret is read, and a PostgreSQL service's database has
   *                    no file to open). OpenRestream: ingest sessions in flight.
@@@ -74,7 +76,9 @@ async function countOne(exec, svc, p, label) {
              for (const field of p.addFields || []) {
                  const raw = at(body, field);
                  // An older service omits the field: count what it does report, never fail the probe.
--                if (raw === undefined || raw === null) continue;
++                if (raw === undefined) continue;
++                // The service reports the field but cannot answer it (null): unknown, never zero.
++                if (raw === null) return { count: null, label, unknown: `unknown at "${field}"` };
                  const extra = Number(raw);
                  if (!Number.isFinite(extra)) return { count: null, label, unknown: `no number at "${field}"` };
                  total += extra;
ddiff --git a/test/cli-inventory.test.js b/test/cli-inventory.test.js
iindex 826c443..bce0bfd 100644
---- a/test/cli-inventory.test.js
++++ b/test/cli-inventory.test.js
@@@ -25,8 +25,9 @@ runTests([
          assert.strictEqual(inv.services.live.socketUnit, 'openvibe-live.socket');
          assert.ok(!inv.services.live.units.includes('openvibe-live.socket'));
          assert.strictEqual(inv.services.live.protected.url, 'http://127.0.0.1:3000/api/streams');
--        // Media: its /api/ready reports ffmpeg and chunked browser recordings in two fields; the probe sums them.
--        assert.deepStrictEqual([inv.services.media.protected.kind, inv.services.media.protected.field, inv.services.media.protected.addFields], ['http-json-count', 'recordings_in_progress', ['chunked_recordings_in_progress']]);
++        // Media: its /api/ready reports ffmpeg recordings, chunked browser recordings, the recordings the
++        // database still holds and the uploads in flight in four fields; the probe sums them.
++        assert.deepStrictEqual([inv.services.media.protected.kind, inv.services.media.protected.field, inv.services.media.protected.addFields], ['http-json-count', 'recordings_in_progress', ['chunked_recordings_in_progress', 'recordings_untracked_in_progress', 'uploads_in_progress']]);
          // Every service on the host is deployed by a strategy (WS-N task 11): Games by pnpm-build.
          assert.ok(Object.values(inv.services).every((s) => s.managed), 'every service is managed');
          assert.deepStrictEqual(Object.fromEntries(['live', 'tools', 'games', 'openre', 'network'].map((id) => [id, inv.services[id].strategy])), { live: 'release-layout', tools: 'multi-app', games: 'pnpm-build', openre: 'release-layout', network: 'git-checkout' });
@@@ -113,7 +114,7 @@ runTests([
          const r = await host.cli('status');
          assert.strictEqual(r.code, 0, r.out);
          assert.match(r.out, /^live +[0-9a-f]{12} ready +openvibe-live=active socket=active live streams=1$/m);
--        assert.match(r.out, /^media .* recordings in progress=2$/m);
++        assert.match(r.out, /^media .* recordings and uploads in progress=2$/m);
          assert.match(r.out, /^tools .*openvibe-tools=active openvibe-tools-maps=active/m);
          assert.match(r.out, /^events .*NOT READY \(502\)/m);
          assert.match(r.out, /^sites .*\(no units\)/m);
ddiff --git a/test/deploy.test.js b/test/deploy.test.js
iindex 8909e1f..35caadb 100644
---- a/test/deploy.test.js
++++ b/test/deploy.test.js
@@@ -128,7 +128,7 @@ runTests([
          host.chunkedRecording = 2;
          const r = await host.cli('deploy', 'media');
          assert.strictEqual(r.code, 5, r.out);
--        assert.match(r.out, /3 recordings in progress — refusing/);
++        assert.match(r.out, /3 recordings and uploads in progress — refusing/);
          assert.strictEqual(host.repo('media').head, from);
          assert.deepStrictEqual(host.restarts(), []);
          assert.ok(host.calls.some((c) => c.cmd === 'curl' && c.args[0] === 'http://127.0.0.1:4100/api/ready'), 'the probe asks Media /api/ready, not PostgreSQL');
@@@ -144,7 +144,7 @@ runTests([
          host.mediaReportChunked = false;   // an older Media: the additive field is absent
          const r = await host.cli('deploy', 'media');
          assert.strictEqual(r.code, 5, r.out);
--        assert.match(r.out, /1 recordings in progress — refusing/);
++        assert.match(r.out, /1 recordings and uploads in progress — refusing/);
          assert.strictEqual(host.repo('media').head, from);
          assert.deepStrictEqual(host.restarts(), []);
      }),
@@@ -156,11 +156,75 @@ runTests([
          host.chunkedRecording = 'many';
          const r = await host.cli('deploy', 'media');
          assert.strictEqual(r.code, 5, r.out);
--        assert.match(r.out, /recordings in progress: unknown \(no number at "chunked_recordings_in_progress"\) — treated as active/);
++        assert.match(r.out, /recordings and uploads in progress: unknown \(no number at "chunked_recordings_in_progress"\) — treated as active/);
          assert.strictEqual(host.repo('media').head, from);
          assert.deepStrictEqual(host.restarts(), []);
      }),
  
++    test('refuses to restart Media while an object upload is in progress', async () => {
++        const host = scenario();
++        const from = host.repo('media').head;
++        push(host, 'media', { 'server/index.js': 'media2();' });
++        host.uploadInProgress = 2;
++        const r = await host.cli('deploy', 'media');
++        assert.strictEqual(r.code, 5, r.out);
++        assert.match(r.out, /2 recordings and uploads in progress — refusing/);
++        assert.strictEqual(host.repo('media').head, from);
++        assert.deepStrictEqual(host.restarts(), []);
++    }),
++
++    test('an unanswerable upload count is unknown (treated as active), never zero', async () => {
++        const host = scenario();
++        const from = host.repo('media').head;
++        push(host, 'media', { 'server/index.js': 'media2();' });
++        host.uploadInProgress = null;   // Media reports the field but cannot answer it
++        const r = await host.cli('deploy', 'media');
++        assert.strictEqual(r.code, 5, r.out);
++        assert.match(r.out, /recordings and uploads in progress: unknown \(unknown at "uploads_in_progress"\) — treated as active/);
++        assert.strictEqual(host.repo('media').head, from);
++        assert.deepStrictEqual(host.restarts(), []);
++    }),
++
++    test('--wait-idle holds the Media restart while an upload is in progress, then deploys', async () => {
++        const host = scenario();
++        const to = push(host, 'media', { 'server/index.js': 'media2();' });
++        let polls = 0;
++        host.http.set('http://127.0.0.1:4100/api/ready', () => {
++            polls += 1;
++            const u = host.units.get('openvibe-media.service');
++            if (!u || u.active !== 'active') return { status: 502 };
++            return { status: 200, body: { status: 'ready', recordings_in_progress: 0, chunked_recordings_in_progress: 0, recordings_untracked_in_progress: 0, uploads_in_progress: polls < 4 ? 1 : 0 } };
++        });
++        const r = await host.cli('deploy', 'media', '--wait-idle');
++        assert.strictEqual(r.code, 0, r.out);
++        assert.match(r.out, /holding the restart/);
++        assert.match(r.out, /idle for 2 checks — proceeding/);
++        assert.strictEqual(host.repo('media').head, to);
++        assert.deepStrictEqual(host.restarts(), ['openvibe-media.service']);
++    }),
++
++    test('--force restarts Media with an upload in progress, loudly, and records it', async () => {
++        const host = scenario();
++        push(host, 'media', { 'server/index.js': 'media2();' });
++        host.uploadInProgress = 1;
++        const r = await host.cli('deploy', 'media', '--force');
++        assert.strictEqual(r.code, 0, r.out);
++        assert.match(r.out, /!!! --force/);
++        assert.match(r.out, /1 recordings and uploads in progress WILL BE DROPPED/);
++        const [rec] = await releases.list(host.exec, host.inv, 'media');
++        assert.strictEqual(rec.forced, true);
++    }),
++
++    test('an older Media that reports no upload counts still deploys', async () => {
++        const host = scenario();
++        const to = push(host, 'media', { 'server/index.js': 'media2();' });
++        host.mediaReportUploads = false;   // the additive fields are absent
++        const r = await host.cli('deploy', 'media');
++        assert.strictEqual(r.code, 0, r.out);
++        assert.strictEqual(host.repo('media').head, to);
++        assert.deepStrictEqual(host.restarts(), ['openvibe-media.service']);
++    }),
++
      test('Media deploys once no recording is running', async () => {
          const host = scenario();
          const to = push(host, 'media', { 'server/index.js': 'media2();' });
ddiff --git a/test/helpers.js b/test/helpers.js
iindex 51852d4..9d51ae9 100644
---- a/test/helpers.js
++++ b/test/helpers.js
@@@ -60,7 +60,7 @@ function inventoryDoc() {
                  env: { required: 'from-example' },
                  port: 4100,
                  ready: { url: 'http://127.0.0.1:4100/healthz', timeoutSeconds: 30 },
--                protected: { kind: 'http-json-count', url: 'http://127.0.0.1:4100/api/ready', field: 'recordings_in_progress', addFields: ['chunked_recordings_in_progress'], label: 'recordings in progress' },
++                protected: { kind: 'http-json-count', url: 'http://127.0.0.1:4100/api/ready', field: 'recordings_in_progress', addFields: ['chunked_recordings_in_progress', 'recordings_untracked_in_progress', 'uploads_in_progress'], label: 'recordings and uploads in progress' },
                  databases: [{ name: 'media', engine: 'postgresql', database: 'ov_media' }],
                  lifecycle: lifecycleDoc({ liveness: { endpoint: '/healthz', means: 'the process answers HTTP and reads its database' }, shutdown: { signal: 'SIGTERM', deadlineSeconds: 70, drains: ['recordings stopped', 'HTTP server closed'] } }),
              },
@@@ -104,7 +104,10 @@ function scenario() {
      host.liveStreams = [];
      host.recording = 0;
      host.chunkedRecording = 0;
++    host.untrackedRecording = 0;
++    host.uploadInProgress = 0;
      host.mediaReportChunked = true;
++    host.mediaReportUploads = true;
      host.sseConnections = 0;
      host.unitRepo = {};
  
@@@ -188,6 +191,10 @@ function scenario() {
          if (!u || u.active !== 'active') return { status: 502 };
          const body = { status: 'ready', recordings_in_progress: host.recording };
          if (host.mediaReportChunked) body.chunked_recordings_in_progress = host.chunkedRecording;
++        if (host.mediaReportUploads) {
++            body.recordings_untracked_in_progress = host.untrackedRecording;
++            body.uploads_in_progress = host.uploadInProgress;
++        }
          return { status: 200, body };
      });
      host.http.set('http://127.0.0.1:4001/api/health', readyFor('openvibe-tools.service'));