openvibers/openvibe.media Commits

Count open browser chunk sessions on /api/ready (#311)

OP OpenVibers committed 8 days ago · 918d5ceb97 · 6 files, +64 −9

A browser MediaRecorder session is a protected recording while chunks keep arriving. /api/ready now reports chunked_recordings_in_progress next to recordings_in_progress; a session silent for five minutes (ten Live slices) is closed, and finalization removes it.

Changes

ddiff --git a/README.md b/README.md
iindex 10d23ba..bf90433 100644
---- a/README.md
++++ b/README.md
@@@ -539,7 +539,7 @@ aggregated and stay queued. The runbook is `docs/cutover-media-billing-readings.
| path | behavior |
|---|---|
| `GET /healthz` | liveness: the process answers (unchanged; it checks nothing else) |
--| `GET /api/ready` | readiness from real checks, each with `status`, `required`, `latency_ms`, `checked_at`. **Required** (503 when one fails): `db` (a real round trip that names the store, postgresql or pglite, and counts the tenants), `valkey` optional (skipped without `VALKEY_URL`), `storage_vods`, `storage_clips`, `storage_pastes`, `storage_thumbnails`, `storage_files`, `storage_objects` (a probe file is written and removed). **Optional** (listed in `degraded`, still 200): `network_jwks`, `remote_b2` / `remote_r2`, `events_outbox`, and `timeline_cache` (bytes vs budget; free disk vs floor). Also `recordings_in_progress`. |
++| `GET /api/ready` | readiness from real checks, each with `status`, `required`, `latency_ms`, `checked_at`. **Required** (503 when one fails): `db` (a real round trip that names the store, postgresql or pglite, and counts the tenants), `valkey` optional (skipped without `VALKEY_URL`), `storage_vods`, `storage_clips`, `storage_pastes`, `storage_thumbnails`, `storage_files`, `storage_objects` (a probe file is written and removed). **Optional** (listed in `degraded`, still 200): `network_jwks`, `remote_b2` / `remote_r2`, `events_outbox`, and `timeline_cache` (bytes vs budget; free disk vs floor). Also `recordings_in_progress` (ffmpeg ingest) and `chunked_recordings_in_progress` (browser MediaRecorder sessions that received a chunk in the last five minutes; additive, so a reader that predates it keeps using the first). |
| `GET /metrics` | Prometheus text (openvibe-shared/metrics) for **direct loopback callers only**; 404 through nginx. HTTP golden signals by route template, process metrics, `release_info`, `release_client_updates_total{outcome,reason}`, plus `media_recordings_in_progress`, `media_timeline_local_bytes`, `media_timeline_local_evictions`, `media_object_locations{provider,state}`, `media_objects{lifecycle_status}`, `media_events_outbox{status}` (only while the outbox runs), `media_job_stale_completions_total{action}` (job writes refused by the lease-token fence). |
| `GET /release.json` | release manifest (`registry.release-manifest@1` 1.1.0, openvibe-shared/release, ADR-016): deployed commit, package versions, components, contract ranges, `metrics_url`. The shared navbar's release-watch polls it. No components are declared, so every release still prompts open tabs to reload. |
| `POST /release-metrics` | open tabs' update reports (release-watch beacons, at most 4 KB; 30 a minute per connecting address, which behind nginx is the proxy, so one budget for all tabs; Sec-GPC/DNT dropped) into `release_client_updates_total` on `/metrics`. 403 in a restore drill, like every write. |
@@@ -741,7 +741,7 @@ Reporting a vulnerability: [SECURITY.md](SECURITY.md). The rules the code keeps:
Production deploys with `sudo ovhost deploy media` on the host (strategy `git-checkout`: fetch,
fast-forward `/opt/openvibe.media`, install on a lockfile change, restart, wait for `/api/ready`, 60 s).
--ovhost refuses the restart while a VOD is recording (`recordings_in_progress` on `/api/ready`); `--wait-idle`
++ovhost refuses the restart while a VOD is recording (`recordings_in_progress` plus `chunked_recordings_in_progress` on `/api/ready`); `--wait-idle`
holds it until none is. The unit is `openvibe-media.service` ([deploy/systemd/](deploy/systemd/openvibe-media.service),
runs as `ubuntu`, media under `/opt/openvibe.media/data`) on `127.0.0.1:4100`, the env file
`/etc/openvibe/media.env`. The listener belongs to `openvibe-media.socket` (with the
ddiff --git a/server/observability.js b/server/observability.js
iindex cde829e..5af378a 100644
---- a/server/observability.js
++++ b/server/observability.js
@@@ -23,6 +23,7 @@ const metrics = require('openvibe-shared/metrics');
const { createReadiness, skip } = require('openvibe-shared/ready');
const copyReport = require('./objects/copy-report');
++const chunkSessions = require('./vod/chunk-sessions');
const OUTBOX_BACKLOG_LIMIT = 1000;
@@@ -176,7 +177,15 @@ function createMediaReadiness({ release, db, config, auth, recorder, events, rem
service: 'media',
release,
checks,
-- details: () => ({ ...(drill ? { mode: 'drill' } : {}), recordings_in_progress: recorder.activeCount() }),
++ // ovhost's protected-session probe sums both: an ffmpeg recording (RTP/RTMP ingest) and an open
++ // browser chunk session are protected sessions alike. The second field is additive, so a Host
++ // that predates it keeps reading recordings_in_progress; a Media that predates it omits it, and
++ // the Host falls back to the first count.
++ details: () => ({
++ ...(drill ? { mode: 'drill' } : {}),
++ recordings_in_progress: recorder.activeCount(),
++ chunked_recordings_in_progress: chunkSessions.countActive(),
++ }),
});
}
ddiff --git a/server/vod/chunk-sessions.js b/server/vod/chunk-sessions.js
nnew file mode 100644
iindex 0000000..db27832
---- /dev/null
++++ b/server/vod/chunk-sessions.js
@@@ -0,0 +1,35 @@
++'use strict';
++/**
++ * OpenVibe.Media — browser MediaRecorder chunk sessions.
++ *
++ * Live's recorder sends a WebM slice every 30 s (`ss.vodRecorder.start(30000)`,
++ * public/js/broadcast.js) and proxies it to POST /api/v1/:app/vods/:id/chunks. A session is open
++ * while chunks keep arriving; finalization removes it, and one that has not received a chunk for
++ * CHUNK_SESSION_IDLE_MS is treated as closed. Five minutes is ten Live slices: a stall (a background
++ * tab whose timer is throttled, a slow network) stays protected, while a browser that vanished
++ * without finalizing stops pinning ovhost's idle wait forever.
++ *
++ * ovhost's Media probe refuses or waits while a recording is in progress, so these sessions must be
++ * counted with the ffmpeg recordings (server/vod/recorder.js activeCount); /api/ready reports both.
++ */
++const CHUNK_SESSION_IDLE_MS = 5 * 60 * 1000;
++
++/** vodId → { filePath, startTime, chunkCount, currentSegmentId, currentSegmentPath, lastChunkAt } */
++const sessions = new Map();
++
++/** Record that a chunk arrived for a session (creation or append). */
++function markChunk(rec, now = Date.now()) {
++ rec.lastChunkAt = now;
++ return rec;
++}
++
++/** Open chunked recordings: not finalized, and a chunk arrived within the idle window. */
++function countActive(now = Date.now()) {
++ let n = 0;
++ for (const rec of sessions.values()) {
++ if (now - (rec.lastChunkAt || 0) <= CHUNK_SESSION_IDLE_MS) n++;
++ }
++ return n;
++}
++
++module.exports = { CHUNK_SESSION_IDLE_MS, sessions, markChunk, countActive };
ddiff --git a/server/vod/routes.js b/server/vod/routes.js
iindex da07356..1f2982e 100644
---- a/server/vod/routes.js
++++ b/server/vod/routes.js
@@@ -22,6 +22,7 @@ const multer = require('multer');
const db = require('../db/database');
const config = require('../config');
const recorder = require('./recorder');
++const chunkSessions = require('./chunk-sessions');
const tools = require('./media-tools');
const { finalizeVod, vodPublic } = require('./finalize');
const { tenantAuth, tenantCors } = require('../auth');
@@@ -61,12 +62,9 @@ const chunkUpload = multer({
},
});
--/**
-- * Active chunked uploads: vodId → { filePath, startTime, chunkCount,
-- * currentSegmentId, currentSegmentPath }. Rebuilt from the DB row if the
-- * server restarted mid-upload.
-- */
--const activeChunkUploads = new Map();
++// Active chunked uploads live in ./chunk-sessions so /api/ready can count them. A session is rebuilt
++// from the DB row if the server restarted mid-upload.
++const activeChunkUploads = chunkSessions.sessions;
/** Private, including legacy rows that have no visibility and is_public = 0. */
function _isPrivate(row) {
@@@ -195,6 +193,7 @@ router.post('/:id/chunks', tenantAuth({ verb: 'write', allowUser: true }), chunk
[filePath, fs.statSync(filePath).size, vod.id]));
rec = { filePath, startTime: Date.now(), chunkCount: 1, currentSegmentId: segmentId, currentSegmentPath: filePath };
++ chunkSessions.markChunk(rec);
activeChunkUploads.set(vod.id, rec);
console.log(`[VOD] Chunked recording started: vod ${vod.id} (${req.appId})`);
@@@ -220,6 +219,7 @@ router.post('/:id/chunks', tenantAuth({ verb: 'write', allowUser: true }), chunk
tools.cleanupTempFile(req.file.path);
rec.chunkCount++;
}
++ chunkSessions.markChunk(rec);
// Update file size and duration estimate in DB
const size = tools.getFileSizeSafe(rec.filePath)
@@@ -426,4 +426,5 @@ router.delete('/:id', tenantAuth({ verb: 'delete' }), async (req, res) => {
});
router.activeChunkUploads = activeChunkUploads;
++router.activeChunkCount = chunkSessions.countActive;
module.exports = router;
ddiff --git a/test/observability.test.js b/test/observability.test.js
iindex 8c90084..d3c1a14 100644
---- a/test/observability.test.js
++++ b/test/observability.test.js
@@@ -62,6 +62,15 @@ const { spawn } = require('child_process');
assert.strictEqual(body.status, 'ready');
assert.strictEqual(body.service, 'media');
assert.strictEqual(body.recordings_in_progress, 1);
++ assert.strictEqual(body.chunked_recordings_in_progress, 0, 'no browser chunk session: the additive field is present at 0');
++ // A chunk session that received a chunk inside the idle window counts; one silent past it does not.
++ const chunkSessions = require('../server/vod/chunk-sessions');
++ chunkSessions.sessions.set(1, { lastChunkAt: Date.now() });
++ chunkSessions.sessions.set(2, { lastChunkAt: Date.now() - chunkSessions.CHUNK_SESSION_IDLE_MS - 1 });
++ r = await request(base, '/api/ready');
++ body = JSON.parse(r.body);
++ assert.deepStrictEqual([body.recordings_in_progress, body.chunked_recordings_in_progress], [1, 1]);
++ chunkSessions.sessions.clear();
const names = Object.keys(body.checks).sort();
assert.deepStrictEqual(names, ['db', 'events_outbox', 'network_jwks', 'object_copies', 'remote_b2', 'remote_r2', 'storage_clips', 'storage_files', 'storage_objects', 'storage_pastes', 'storage_thumbnails', 'storage_vods', 'timeline_cache', 'valkey']);
// r2 is not configured: listed as skipped with the reason, never a pretend pass (WS-Q task 7), and an optional
ddiff --git a/test/recordings.test.js b/test/recordings.test.js
iindex a66fb40..6c16e7c 100644
---- a/test/recordings.test.js
++++ b/test/recordings.test.js
@@@ -95,6 +95,7 @@ const express = require('express');
const f = new FormData();
f.append('chunk', new Blob([Buffer.alloc(40000, 1)], { type: 'video/webm' }), 'a.webm');
await call('POST', `/api/v1/live/vods/${r.body.id}/chunks`, f);
++ assert.strictEqual(require('../server/vod/chunk-sessions').countActive(), made.length + 1, 'a browser chunk is an open protected session');
const objectId = (await finalizeVod(r.body.id)).object_id;
const obj = await model.getObject(objectId);
const bytes = {};