openvibers/openvibe.media Commits

Media: the per-tenant job cap counts one lane (paused transcodes no longer refuse thumbnails and hashes) (#103)

OP OpenVibers committed 13 Oct 2026 · d9c46b420d · 3 files, +72 −5

On 2026-10-13 every new recording's thumbnail, content hash and AI timeline was refused for hours with `429 media.job.too_many`. The refusals showed up in Live visitors' consoles and failed Live's browser check.

**Cause**
- The worker holds the **heavy** lane (`object.cmaf`, `object.pack`, splits, renditions) while any recording runs, by design: the 4-core host's CPU stays with the recordings.
- Streams ran back to back from 14:52, so 30 `object.cmaf` and 20 `object.pack` jobs sat queued, all due and never claimed. The last one finished at 15:11.
- The per-tenant cap of 50 counted every lane together, so those 50 paused transcodes held all of it.
- Production log: `[HLS] Queue failed … At most 50 jobs`, `[Jobs] could not schedule object.hash for live`, `[VOD] Timeline queue failed for vod 5729`.

**Fix** (`server/jobs/queue.js`)
- The cap now counts **one lane**: queued and running jobs of the tenant whose type runs in the same lane as the new one (new `typesInLane`). Each lane holds at most 50 per tenant.
- A backlog of transcodes waiting out the streams fills only its own lane; thumbnails, hashes, finalize and clip cuts keep theirs.
- The 429 message names the lane. `docs/object-model.md` says so.

**Tests:** `test/job-cap-per-lane.test.js`:
- transcodes are heavy and thumbnails light;
- 50 queued heavy jobs refuse the 51st (429, "heavy jobs") but not a light job;
- another tenant is unaffected;
- a proposal is not capped;
- the light lane has a cap of its own.

`test/jobs.test.js` passes unchanged.

Changes

ddiff --git a/docs/object-model.md b/docs/object-model.md
iindex e2ef99f..184c22e 100644
---- a/docs/object-model.md
++++ b/docs/object-model.md
@@@ -671,7 +671,8 @@ that needs them GETs the job with its own tenant-scoped token (`GET /api/v2/:app
**Idempotency.** `Idempotency-Key` (or `idempotency_key`) is unique per tenant: the same key with the
same request (type, object, params) answers with the job it made (`Idempotent-Replayed: true`); a
different request under a used key is 409 `media.job.idempotency_conflict`. At most 50 jobs per tenant
--may be queued or running (429 `media.job.too_many`).
++may be queued or running **in each lane** (429 `media.job.too_many`): the heavy lane waits while a recording runs,
++so a backlog of transcodes never stops a thumbnail, a content hash or a clip.
**Cancellation, by the owner.** A proposed or queued job is cancelled at once. A running job gets
`cancel_requested`; the worker aborts it (its ffmpeg is killed) and it ends `cancelled`. Finished
ddiff --git a/server/jobs/queue.js b/server/jobs/queue.js
iindex 9ed3b5a..a188632 100644
---- a/server/jobs/queue.js
++++ b/server/jobs/queue.js
@@@ -35,7 +35,10 @@ const events = require('../events');
const STATUSES = ['proposed', 'queued', 'running', 'succeeded', 'failed', 'cancelled'];
const FINISHED = ['succeeded', 'failed', 'cancelled'];
const ACTIVE = ['queued', 'running'];
--const MAX_ACTIVE_PER_TENANT = 50; // queued + running jobs one tenant may have (proposals excluded)
++// Queued + running jobs one tenant may have IN ONE LANE (proposals excluded). Per lane: the heavy lane waits while a
++// recording runs (worker.js), and on 2026-10-13 a day of back-to-back streams left 50 paused transcodes holding a single
++// tenant-wide cap, so every thumbnail and content hash of the new recordings was refused (429) for hours.
++const MAX_ACTIVE_PER_TENANT = 50;
// Service-wide maintenance jobs (storage.orphans.scan) belong to no tenant: they run under this app id,
// which has no apps row, so no API credential reaches them, and they announce no events.
const SYSTEM_APP = '_media';
@@@ -75,6 +78,11 @@ function typeSpec(type) {
loadBuiltins();
return TYPES.get(type) || null;
}
++/** The job types that run in `lane` (the per-tenant cap counts one lane). */
++function typesInLane(lane) {
++ loadBuiltins();
++ return [...TYPES.values()].filter((s) => s.lane === lane).map((s) => s.type);
++}
function typeNames() {
loadBuiltins();
return [...TYPES.keys()];
@@@ -266,9 +274,11 @@ async function enqueue({ appId, type, objectId = null, params = {}, status = 'qu
if (same) { out = { job: same, created: false, replayed: false, deduped: true }; return; }
}
if (status === 'queued') {
-- const active = (await db.get("SELECT COUNT(*) AS n FROM media_jobs WHERE app_id = ? AND status IN ('queued', 'running')", [appId])).n;
++ const lane = typesInLane(spec.lane);
++ const active = (await db.get(`SELECT COUNT(*) AS n FROM media_jobs WHERE app_id = ? AND status IN ('queued', 'running')
++ AND job_type IN (${lane.map(() => '?').join(', ')})`, [appId, ...lane])).n;
if (active >= MAX_ACTIVE_PER_TENANT) {
-- throw new JobError('media.job.too_many', `At most ${MAX_ACTIVE_PER_TENANT} jobs may be queued or running per tenant`, { status: 429, retryAfterS: 60 });
++ throw new JobError('media.job.too_many', `At most ${MAX_ACTIVE_PER_TENANT} ${spec.lane} jobs may be queued or running per tenant`, { status: 429, retryAfterS: 60 });
}
}
await db.run(`INSERT INTO media_jobs (id, app_id, object_id, job_type, status, idempotency_key, request_hash, params, max_attempts,
@@@ -488,7 +498,7 @@ async function prune({ days = 30, types = ['thumbnail.regenerate', 'object.hash'
module.exports = {
STATUSES, FINISHED, ACTIVE, MAX_ACTIVE_PER_TENANT, SYSTEM_APP, JobError, bus,
-- register, typeSpec, typeNames, requestHash, parseJson,
++ register, typeSpec, typeNames, typesInLane, requestHash, parseJson,
get, getForApp, jobPublic, jobEvent, list, counts,
enqueue, approve, cancel,
claim, renew, saveCheckpoint, succeed, fail, markCancelled, recoverInterrupted, waitFor, prune, staleStats,
ddiff --git a/test/job-cap-per-lane.test.js b/test/job-cap-per-lane.test.js
nnew file mode 100644
iindex 0000000..bc7d370
---- /dev/null
++++ b/test/job-cap-per-lane.test.js
@@@ -0,0 +1,56 @@
++'use strict';
++// The per-tenant job cap counts one lane. On 2026-10-13 the heavy lane waited out a day of back-to-back recordings
++// (worker.js runs no transcode while a recording does) with 30 object.cmaf and 20 object.pack jobs queued: they held the
++// whole tenant-wide cap of 50, so every thumbnail and content hash of the new recordings was refused with 429 for hours.
++const assert = require('assert');
++const fs = require('fs');
++const os = require('os');
++const path = require('path');
++const http = require('http');
++
++const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-jobcap-'));
++const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
++Object.assign(process.env, {
++ VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
++ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),
++ MEDIA_INVARIANT_SCAN_HOURS: '0',
++});
++
++(async () => {
++ const stub = http.createServer((req, res) => {
++ res.setHeader('Content-Type', 'application/json');
++ if (req.url === '/oauth/token') return res.end(JSON.stringify({ access_token: 'tok', token_type: 'Bearer', expires_in: 300 }));
++ res.statusCode = 404; res.end('{}');
++ });
++ await new Promise((r) => stub.listen(0, '127.0.0.1', r));
++ const base = `http://127.0.0.1:${stub.address().port}`;
++ const db = require('../server/db/database');
++ await db.upsertApp({ app_id: 'live', api_key: 'live-key-jobcap' });
++ await db.upsertApp({ app_id: 'games', api_key: 'games-key-jobcap' });
++ require('../server/events').init({ eventsUrl: base, clientSecret: 's', networkUrl: base, intervalMs: 60000 });
++ const queue = require('../server/jobs/queue');
++ // Nothing runs these: no worker is started, so they stay queued like the paused heavy lane.
++ queue.register('test.cap.heavy', { lane: 'heavy', run: async () => ({}) });
++ queue.register('test.cap.light', { lane: 'light', run: async () => ({}) });
++ const cap = queue.MAX_ACTIVE_PER_TENANT;
++
++ assert.ok(queue.typesInLane('heavy').includes('object.cmaf') && queue.typesInLane('heavy').includes('object.pack'), 'transcodes are heavy');
++ assert.ok(queue.typesInLane('light').includes('thumbnail.regenerate') && !queue.typesInLane('light').includes('object.cmaf'), 'thumbnails are light');
++
++ for (let i = 0; i < cap; i++) await queue.enqueue({ appId: 'live', type: 'test.cap.heavy', params: { n: i } });
++ await assert.rejects(queue.enqueue({ appId: 'live', type: 'test.cap.heavy', params: { n: cap } }),
++ (e) => e.code === 'media.job.too_many' && e.status === 429 && /heavy jobs/.test(e.message), 'the heavy lane is full');
++ const light = await queue.enqueue({ appId: 'live', type: 'test.cap.light', params: { n: 0 } });
++ assert.ok(light.created, 'a full heavy lane does not stop a light job');
++ assert.ok((await queue.enqueue({ appId: 'games', type: 'test.cap.heavy', params: { n: 0 } })).created, 'another tenant has its own lanes');
++ // A proposal is not counted and is still accepted with a full lane.
++ assert.ok((await queue.enqueue({ appId: 'live', type: 'test.cap.heavy', params: { n: 'p' }, status: 'proposed' })).created, 'a proposal is not capped');
++
++ for (let i = 1; i < cap; i++) await queue.enqueue({ appId: 'live', type: 'test.cap.light', params: { n: i } });
++ await assert.rejects(queue.enqueue({ appId: 'live', type: 'test.cap.light', params: { n: cap } }),
++ (e) => e.code === 'media.job.too_many' && /light jobs/.test(e.message), 'the light lane has the same cap of its own');
++
++ console.log('job-cap-per-lane: ok');
++ stub.close();
++ process.exit(0);
++})().catch((err) => { console.error(err); process.exit(1); });