openvibers/openvibe.media Commits

Queue the CMAF timeline at finalize and sign playlists for longer

OP OpenVibers committed 4 Oct 2026 · 76f859053b · 10 files, +327 −15

Asked: T4 step 3 (F3.c) — OpenVibe.Media: the CMAF timeline gets queued automatically, and signed playlists stay valid longer than one download token. Security-relevant: it changes signed-URL tokens for private media. Open ONE pull request from current origin/main (git fetch first; read with `git show ori…
Harness job media-t4-step-3-f3-c-211950 (plan T4).

Changes

ddiff --git a/.env.example b/.env.example
iindex 9ddc9e3..cd64a99 100644
---- a/.env.example
++++ b/.env.example
@@@ -140,3 +140,6 @@ MEDIA_CAPABILITY_GATE=report
# 1 = the object.cmaf job (CMAF segments of a finished video, indexed in media_timeline) and the HLS routes
# /o/:id/master.m3u8, /o/:id/source/… ; off (default): the job is refused and the routes 404, nothing else changes.
#MEDIA_HLS_ENABLED=0
++# Lifetime in seconds of a signed HLS playlist (download?format=json → hls_url of a private or sandbox object; its own
++# signature purpose, accepted only by the HLS routes). Default 21600 (6 h), clamped to 60..43200.
++#MEDIA_HLS_PLAYLIST_TTL_S=21600
ddiff --git a/README.md b/README.md
iindex d81c6ed..6427256 100644
---- a/README.md
++++ b/README.md
@@@ -458,7 +458,7 @@ Counters are per process (a restart forgets them). `test/actor-limits.test.js`.
| `GET /live/:sel/transcript.json` | **transcript + AI timeline API** — full audio-transcription log and AI overview timeline. `:sel` = slot id / slot slug → slot-scoped sessions; **`@username`** → user-scoped (all their slots, works while offline). Returns `{ live, current, sessions[] (each: ai_overview, transcript, duration…), streamer (overview + stream memories), user }`. `?limit=1..50` sessions (default 10), `?app=`. Cached **30s** (that's the rate limit), CORS-open. |
| `GET /live/:sel/chat-insight.json` | **chat insight API** — a user's chat-related AI insight/timeline (`:sel` = `@username` or numeric user id): today-vs-alltime chat overviews, condensed memory, event timeline, plus their streamer overview + stream memories when they stream. Proxied from the app's public chat-AI API over loopback; cached **30s**, CORS-open. |
| `GET /v/:id/transcript.json` | **VOD transcript API** — transcript + AI overview for one existing VOD id (`{ vod_id, title, duration_seconds, ai_overview, transcript, ai_analyzed_at }`). Private VODs → 404. Cached 30s, CORS-open. |
--| `GET /o/:id/master.m3u8`, `/o/:id/source/index.m3u8`, `/o/:id/source/{init.mp4,NNNNNN.m4s}` | segment-native video (F3.1), only with `MEDIA_HLS_ENABLED=1` (off by default; off, these 404): HLS playlists written from the object's `media_timeline` rows (VOD, `#EXT-X-ENDLIST`) and its CMAF segments (Range; a segment packed by `object.pack` (F3.3) is a byte range of its chunk). Same check as `GET /o/:id` (and, like it, no `Access-Control-Allow-Origin` on a private or sandbox answer): private and sandbox objects only with the object's `?exp&sig` (`/api/v2/:app/objects/:id/download?format=json` → `hls_url`), carried onto every URI; deleted 410. The `object.cmaf` job makes the timeline. `noindex` |
++| `GET /o/:id/master.m3u8`, `/o/:id/source/index.m3u8`, `/o/:id/source/{init.mp4,NNNNNN.m4s}` | segment-native video (F3.1), only with `MEDIA_HLS_ENABLED=1` (off by default; off, these 404): HLS playlists written from the object's `media_timeline` rows (VOD, `#EXT-X-ENDLIST`) and its CMAF segments (Range; a segment packed by `object.pack` (F3.3) is a byte range of its chunk). Same check as `GET /o/:id` (and, like it, no `Access-Control-Allow-Origin` on a private or sandbox answer): private and sandbox objects only with the object's `?exp&sig` (`/api/v2/:app/objects/:id/download?format=json` → `hls_url`, a playlist token valid `MEDIA_HLS_PLAYLIST_TTL_S` (6 h; 60 s to 12 h) that no other route accepts; a download signature works too), carried onto every URI; deleted 410. The `object.cmaf` job makes the timeline; a recording's finalize queues it, and `download?format=json` queues it for an object that has none yet. `noindex` |
| `GET /robots.txt` | crawler policy (`openvibe-shared/seo.robotsTxt`: AI and search crawlers named, `/api/`, `/auth/`, `/internal/`, `/o/`, `/metrics`, `/live/`, `/me` disallowed) and the sitemap |
| `GET /sitemap.xml` | the media index and the watch pages Media is the canonical home of: public, playable VODs and clips of apps other than Live (Live lists its own `/vod` and `/clip` pages), never AI clips, private, unlisted, sandbox, recording or failed items; thumbnails as image entries. Every listed page renders `index, follow` |
| `GET /llms.txt` | a map of the site for language-model crawlers |
ddiff --git a/docs/media-fabric.md b/docs/media-fabric.md
iindex 897cbdf..34d2a05 100644
---- a/docs/media-fabric.md
++++ b/docs/media-fabric.md
@@@ -134,11 +134,22 @@ med_xyz/{captions/*.vtt, storyboard.webp+.vtt, waveform, metadata}
form over chunk URIs is not used, so URIs, signed playlists and caches stay as they were. Purge, VOD delete, holds
and the orphan report see a chunk through the rows that name it (each location deleted once, every row kept while its
chunk's delete fails). All of it behind `MEDIA_HLS_ENABLED`; no schema change (F3.1's columns carry it).
++- **F3.c, shipped: the timeline queued by itself, signed playlists, the MP4 fallback.** A recording's finalize queues
++ `object.cmaf` for its object once it is ready (`server/objects/timeline-queue.js`; `dedupeActive` and the idempotency
++ key `object.cmaf:<id>`, so a second finalize joins the same job; a queue error is logged, never fails the finalize),
++ and `GET …/download?format=json` queues it lazily for a media object without a timeline (a VOD older than F3.1):
++ `hls_url` appears once the job ran. Nothing is queued with `MEDIA_HLS_ENABLED` off, for a non-media object or once
++ the source timeline exists. For a private or sandbox object `hls_url` is a **playlist token**: its own MAC purpose
++ (`hls`), valid for `MEDIA_HLS_PLAYLIST_TTL_S` (6 h by default, 60 s to 12 h) so a viewer plays a long VOD on one page
++ load, carried onto every playlist and segment URI, and accepted only by the HLS routes (never by `GET /o/:id` or any
++ other route); the HLS routes still accept the object's download signature, so earlier URLs keep working. A public
++ object's `hls_url` stays unsigned. `GET …/download?format=mp4` answers the object's `remux` variant (`object.remux`,
++ `-movflags +faststart` for an MP4 source; the container stays the source's) as a signed URL, or queues the remux
++ (`dedupeActive`) and answers `202 { job_id }` while it is missing.
**Still open in F3:** the growing live/DVR playlist written as OpenRe segments (F3.2), write-behind durability for
live segments and its upload-lag metric, virtual and materialized clips over the timeline (chunk reference counts),
-- sprites and captions on it, the faststart MP4 fallback as a derivative, segment-bucket demand, a finalize hook that
-- queues `object.cmaf` and `object.pack` by itself, signed playlists that outlive one signed-URL lifetime, and Live's
-- player moving to HLS. Renditions are F4.
++ sprites and captions on it, queuing `object.pack` after the cut by itself, segment-bucket demand, and Live's player
++ moving to HLS. Renditions are F4.
## 4. Clips reuse the source
@@@ -290,9 +301,10 @@ metric. Budgets per class and provider with a forecast; `media.provider.cost_thr
and per-class monthly R2 storage ceilings on its moves (F2.6, shipped), the move cleanup job and its alert after
three failures (§6, shipped). Still open in F2: every class (not only native objects ↔ R2) through the one sweep.
- **F3 segment-native video:** the timeline index and the CMAF/HLS source representation of finished video (F3.1,
-- shipped; §3), packing into ~60 s chunks (F3.3, shipped; §3); still open: live DVR from OpenRe (CMAF/HLS recording
-- with the growing playlist), virtual clips, sprites and captions on the timeline, MP4 fallback; Live's player moves to
-- HLS.
++ shipped; §3), packing into ~60 s chunks (F3.3, shipped; §3), the timeline queued at finalize (and lazily on
++ download), signed playlists outliving a download URL and the faststart MP4 fallback (F3.c, shipped; §3); still open:
++ live DVR from OpenRe (CMAF/HLS recording with the growing playlist), virtual clips, sprites and captions on the
++ timeline, segment-bucket demand; Live's player moves to HLS.
- **F4 reactive derivatives:** on-demand renditions and image variants, keep-vs-regenerate economics, AV1 for viral
VODs, compute placement.
- **F5 multi-CDN delivery:** Bunny path (B2 origin), R2 custom-domain path, OpenVibe edge first up to capacity, route
ddiff --git a/docs/object-model.md b/docs/object-model.md
iindex 5ce0219..a81c76c 100644
---- a/docs/object-model.md
++++ b/docs/object-model.md
@@@ -222,7 +222,7 @@ All routes live under `/api/v2/:app/objects`.
| GET | `/` | cursor list, newest first. `?limit (≤200)&cursor&namespace (and below)&kind&visibility&status&owner (usr_…)&user_id&include_deleted` returns `{ objects, next_cursor }`. A Network token sees only the namespaces it may list |
| DELETE | `/:id` | soft delete: `lifecycle_status = deleted`, and the bytes are kept for `MEDIA_DELETE_RETENTION_DAYS` (default 30). 409 `media.object.held` under a hold. 409 `media.object.legacy_managed` for projected objects, which are deleted through their v1 route |
| POST | `/:id/restore` | undo a soft delete within the retention period (410 once purged) |
--| GET | `/:id/download` | public or unlisted: 302 to the public location (the inherited URL for projected objects such as `/v/:id`, `/c/:id`, `/f/:key`, `/t/:name`, `/p/:slug/screenshot`; otherwise `/o/:id`). Private: `{ url, expires_at }` signed for `?ttl` seconds (30 to 3600, default `MEDIA_SIGNED_URL_TTL_S` = 300), or a 302 with `?redirect=1`. `?format=json` always answers JSON. 410 when deleted |
++| GET | `/:id/download` | public or unlisted: 302 to the public location (the inherited URL for projected objects such as `/v/:id`, `/c/:id`, `/f/:key`, `/t/:name`, `/p/:slug/screenshot`; otherwise `/o/:id`). Private: `{ url, expires_at }` signed for `?ttl` seconds (30 to 3600, default `MEDIA_SIGNED_URL_TTL_S` = 300), or a 302 with `?redirect=1`. `?format=json` always answers JSON (with `MEDIA_HLS_ENABLED`, `hls_url` and, when private, `hls_expires_at`; docs/media-fabric.md §3). `?format=mp4`: the faststart `remux` variant as a signed `{ url, expires_at, object_id }`, or `202 { job_id }` while `object.remux` makes it. 410 when deleted |
| GET | `/:id/holds` | active holds (`?all=1` includes released ones), `inherited_holds` (a clip's VOD's) and `held` |
| POST | `/:id/holds` | **app key only.** `{ kind, reason, note, placed_by \| created_by }` |
| DELETE | `/:id/holds/:holdId` | **app key only.** Releases the hold (`released_at`, `released_by`) |
ddiff --git a/server/config.js b/server/config.js
iindex eced9c1..31c5859 100644
---- a/server/config.js
++++ b/server/config.js
@@@ -89,6 +89,8 @@ const config = {
// media_timeline (/o/:id/master.m3u8, /o/:id/source/…). Off by default: off, the job is refused and the routes 404.
hls: {
enabled: ['1', 'true', 'on'].includes(String(process.env.MEDIA_HLS_ENABLED || '').toLowerCase()),
++ // Lifetime of a signed playlist (purpose 'hls', accepted only by the HLS routes): 60 s to 12 h, 6 h by default.
++ playlistTtlS: Math.min(43200, Math.max(60, intEnv('MEDIA_HLS_PLAYLIST_TTL_S', 21600))),
},
jobs: {
ddiff --git a/server/objects/routes.js b/server/objects/routes.js
iindex ec7cc70..7819a69 100644
---- a/server/objects/routes.js
++++ b/server/objects/routes.js
@@@ -665,21 +665,59 @@ router.get('/:id/download', read, limits('media.object.download'), async (req, r
if (!obj) return;
if (obj.lifecycle_status === 'deleted') return problem(res, 410, 'media.object.deleted', 'Object was deleted');
if (obj.lifecycle_status !== 'ready') return problem(res, 409, 'media.object.not_ready', `Object is ${obj.lifecycle_status}`);
-- const json = req.query.format === 'json';
res.set('Cache-Control', cache.htmlHeaders({ private: true }));
++ if (req.query.format === 'mp4') return await downloadMp4(req, res, obj);
++ const json = req.query.format === 'json';
// Developer-project sandbox objects are never public, whatever their visibility: always signed.
// With segment-native video on (MEDIA_HLS_ENABLED) and a timeline made, the JSON answer also names the HLS master
-- // playlist: open for a public object, under the same signature as `url` for a private one.
++ // playlist: open for a public object, signed as a playlist (purpose 'hls', MEDIA_HLS_PLAYLIST_TTL_S) for a private
++ // one. A media object without a timeline yet (a VOD older than F3.1) gets object.cmaf queued, and hls_url once it ran.
const hls = json && config.hls.enabled && await require('./timeline').has(obj.id);
++ if (json && config.hls.enabled && !hls) {
++ try { await require('./timeline-queue').queueCmaf(obj.app_id, obj.id); } catch (err) { console.warn(`[Objects] Timeline queue failed for ${obj.id}:`, err.message); }
++ }
if (obj.visibility !== 'private' && !await db.isSandboxTenant(obj.app_id)) {
const url = model.legacyPublicUrl(obj) || `${config.publicUrl}/o/${obj.id}`;
return json ? res.json({ url, expires_at: null, public: true, ...(hls && { hls_url: `${config.publicUrl}/o/${obj.id}/master.m3u8` }) }) : res.redirect(302, url);
}
const signed = signing.signedDownloadUrl(obj.id, req.query.ttl);
if (['1', 'true'].includes(String(req.query.redirect || ''))) return res.redirect(302, signed.url);
-- res.json({ url: signed.url, expires_at: signed.expires_at, public: false, ...(hls && { hls_url: signed.url.replace('?', '/master.m3u8?') }) });
++ const playlist = hls && signing.signedPlaylistUrl(obj.id);
++ res.json({ url: signed.url, expires_at: signed.expires_at, public: false, ...(playlist && { hls_url: playlist.url, hls_expires_at: playlist.expires_at }) });
});
++/**
++ * GET …/download?format=mp4: the object's faststart MP4 (its `remux` variant, object.remux) as a signed URL (JSON, or a
++ * 302 with ?redirect=1). Without the variant yet: object.remux is queued (joining one already queued or running) and
++ * the answer is 202 { job_id }; ask again once the job succeeded.
++ */
++async function downloadMp4(req, res, obj) {
++ if (!require('../jobs/derive').isMediaObject(obj)) return problem(res, 400, 'media.object.not_media', 'Only video and audio objects have an MP4 download');
++ const variant = await model.getVariant(obj.id, 'remux');
++ const mp4 = variant && await model.getObject(variant.derived_object_id);
++ if (mp4 && mp4.lifecycle_status === 'ready') {
++ const signed = signing.signedDownloadUrl(mp4.id, req.query.ttl);
++ if (['1', 'true'].includes(String(req.query.redirect || ''))) return res.redirect(302, signed.url);
++ return res.json({ url: signed.url, expires_at: signed.expires_at, public: false, object_id: mp4.id });
++ }
++ const queue = require('../jobs/queue');
++ try {
++ const r = await queue.enqueue({
++ appId: obj.app_id, type: 'object.remux', objectId: obj.id, params: {}, dedupeActive: true,
++ createdBy: 'system:download', ownerUserId: req.authType === 'user' ? req.userId : null,
++ });
++ if (r.created) require('../jobs/worker').kick();
++ res.status(202).json({ job_id: r.job.id });
++ } catch (err) {
++ if (!(err instanceof queue.JobError)) {
++ console.error('[Objects] MP4 remux queue error:', err.message);
++ return problem(res, 500, 'media.job.failed_request', 'Could not queue the MP4');
++ }
++ if (err.retryAfterS) res.set('Retry-After', String(err.retryAfterS));
++ problem(res, err.status || 400, err.code, err.message);
++ }
++}
++
// ── Retention holds ──────────────────────────────────────────
router.get('/:id/holds', read, limits('media.object.holds'), async (req, res) => {
@@@ -736,7 +774,8 @@ const publicRouter = express.Router();
// GET /o/:id/master.m3u8, /o/:id/source/index.m3u8 and /o/:id/source/{init.mp4,NNNNNN.m4s}: the playlists are written
// from the object's media_timeline rows (objects/timeline.js), never read from files. Every one passes the same check
// as GET /o/:id: public and unlisted objects openly, private and sandbox ones only with the object's valid ?exp&sig
--// (from /download), which a signed playlist carries onto each URI; deleted 410, not ready 404. Off, these paths 404.
++// (a playlist token from /download's hls_url, or a download signature), which a signed playlist carries onto each URI;
++// deleted 410, not ready 404. Off, these paths 404.
// Like GET /o/:id, a private or sandbox answer carries no Access-Control-Allow-Origin; a public one may be read cross-origin.
// A segment's copy is the placement router's choice; a packed one (F3.3) is a ranged read of its ~60 s chunk.
const HLS_TYPE = 'application/vnd.apple.mpegurl';
@@@ -749,7 +788,8 @@ async function hlsObject(req, res, next) {
if (require('../drill').refuseBytes(res)) return null;
const obj = await model.getObject(String(req.params.id || ''));
if (!obj) { res.status(404).json({ error: 'Not found' }); return null; }
-- const signed = !!req.query.sig && signing.verifyDownload(obj.id, req.query.exp, req.query.sig);
++ // A playlist token (purpose 'hls', from /download's hls_url) or, as before, the object's download signature.
++ const signed = !!req.query.sig && (signing.verifyPlaylist(obj.id, req.query.exp, req.query.sig) || signing.verifyDownload(obj.id, req.query.exp, req.query.sig));
const closed = obj.visibility === 'private' || await db.isSandboxTenant(obj.app_id);
if (closed && !signed) { res.status(404).json({ error: 'Not found' }); return null; }
if (obj.lifecycle_status === 'deleted') { res.status(410).json({ error: 'Gone' }); return null; }
ddiff --git a/server/objects/signing.js b/server/objects/signing.js
iindex 088fd17..4acaefb 100644
---- a/server/objects/signing.js
++++ b/server/objects/signing.js
@@@ -40,6 +40,8 @@ function safeEqual(a, b) {
function nowS() { return Math.floor(Date.now() / 1000); }
++const PLAYLIST_MAX_S = 12 * 3600;
++
/** Short-lived GET URL for /o/:id. ttl is clamped to [30 s, 1 h]. */
function signedDownloadUrl(objectId, ttlS = config.objects.signedUrlTtlS) {
const exp = nowS() + Math.min(3600, Math.max(30, Number(ttlS) || config.objects.signedUrlTtlS));
@@@ -55,6 +57,25 @@ function verifyDownload(objectId, exp, sig) {
return safeEqual(sig, mac('get', objectId, e));
}
++/**
++ * Signed HLS master playlist of an object (/o/:id/master.m3u8), carried onto every playlist and segment URI under it.
++ * Its own purpose ('hls'): a viewer plays a long VOD on one page load, so it lives longer than a download URL
++ * (MEDIA_HLS_PLAYLIST_TTL_S, 60 s to 12 h), and only the HLS routes accept it — never GET /o/:id or any other route.
++ */
++function signedPlaylistUrl(objectId, ttlS = config.hls.playlistTtlS) {
++ const exp = nowS() + Math.min(PLAYLIST_MAX_S, Math.max(60, Number(ttlS) || config.hls.playlistTtlS));
++ return {
++ url: `${config.publicUrl}/o/${encodeURIComponent(objectId)}/master.m3u8?exp=${exp}&sig=${mac('hls', objectId, exp)}`,
++ expires_at: new Date(exp * 1000).toISOString(),
++ };
++}
++
++function verifyPlaylist(objectId, exp, sig) {
++ const e = Number(exp);
++ if (!Number.isInteger(e) || e < nowS() || e > nowS() + PLAYLIST_MAX_S + 60) return false;
++ return safeEqual(sig, mac('hls', objectId, e));
++}
++
/**
* Short-lived GET URL for a v1 file (/f/:key). Its own purpose ('getf') and id space (file:<key>), so
* a file signature is never valid for an object or the other way round. Used for developer-project
@@@ -159,6 +180,6 @@ function verifyMultipartToken(token, { tenant, objectId, uploadId, totalSize })
}
module.exports = {
-- signedDownloadUrl, verifyDownload, signedFileUrl, verifyFile, signedMediaUrl, verifyMedia,
++ signedDownloadUrl, verifyDownload, signedPlaylistUrl, verifyPlaylist, signedFileUrl, verifyFile, signedMediaUrl, verifyMedia,
uploadToken, uploadTtl, checkUploadToken, verifyUploadToken, multipartToken, verifyMultipartToken,
};
ddiff --git a/server/objects/timeline-queue.js b/server/objects/timeline-queue.js
nnew file mode 100644
iindex 0000000..2080682
---- /dev/null
++++ b/server/objects/timeline-queue.js
@@@ -0,0 +1,29 @@
++/**
++ * OpenVibe.Media — queue the timeline of a finished video (docs/media-fabric.md §3, F3.c)
++ *
++ * queueCmaf(appId, objectId) enqueues `object.cmaf` for a ready media object that has no source timeline yet: from the
++ * finalize of a recording (server/vod/finalize.js), and lazily from GET …/download?format=json so a VOD older than F3.1
++ * gets one on its first request. Nothing happens (null) with MEDIA_HLS_ENABLED off, for a missing, unready or non-media
++ * object, or once the timeline exists. One job per object: dedupeActive joins a queued or running cut, and the
++ * idempotency key `object.cmaf:<id>` answers a later call with the same job. → the job id, or null.
++ */
++'use strict';
++
++const config = require('../config');
++
++async function queueCmaf(appId, objectId, { segment_seconds = null } = {}) {
++ if (!config.hls.enabled || !appId || !objectId) return null;
++ const obj = await require('./model').getObject(objectId);
++ if (!obj || obj.lifecycle_status !== 'ready' || !require('../jobs/derive').isMediaObject(obj)) return null;
++ if (await require('./timeline').has(obj.id)) return null;
++ const queue = require('../jobs/queue');
++ const params = require('../jobs/cmaf').spec.validate({ appId, obj, params: segment_seconds != null ? { segment_seconds } : {} });
++ const r = await queue.enqueue({
++ appId, type: 'object.cmaf', objectId: obj.id, params,
++ dedupeActive: true, idempotencyKey: `object.cmaf:${obj.id}`, createdBy: 'system:timeline',
++ });
++ if (r.created) require('../jobs/worker').kick();
++ return r.job ? r.job.id : null;
++}
++
++module.exports = { queueCmaf };
ddiff --git a/server/vod/finalize.js b/server/vod/finalize.js
iindex 084efb1..99a9a15 100644
---- a/server/vod/finalize.js
++++ b/server/vod/finalize.js
@@@ -385,8 +385,16 @@ async function _doFinalize(vodId, opts) {
[stored, durationSource, stat.size, measured, probeFormatJson, 'ok', JSON.stringify(issues), vodId])),
payload: async () => await vodPublic(await db.getVodById(vodId)),
});
++ const ready = await db.getVodById(vodId);
++ // Segment-native video (MEDIA_HLS_ENABLED): the finished recording's timeline is cut by object.cmaf. A queue error
++ // never fails the finalize; GET …/download?format=json queues it again on the first request.
++ try {
++ if (ready && ready.object_id) await require('../objects/timeline-queue').queueCmaf(vod.app_id, ready.object_id);
++ } catch (err) {
++ console.warn(`[VOD] Timeline queue failed for vod ${vodId}:`, err.message);
++ }
console.log(`[VOD] Finalized: vod ${vodId}, ${stored}s (${durationSource}), ${(stat.size / 1024 / 1024).toFixed(1)}MB`);
-- return await db.getVodById(vodId);
++ return ready;
}
module.exports = { finalizeVod, isFinalizing, vodPublic, rebuildWebmFromMaster, _absUrl, FINALIZE_ISSUES };
ddiff --git a/test/timeline-queue.test.js b/test/timeline-queue.test.js
nnew file mode 100644
iindex 0000000..5e3ceac
---- /dev/null
++++ b/test/timeline-queue.test.js
@@@ -0,0 +1,197 @@
++'use strict';
++// The timeline queued by itself and signed playlists (F3.c; docs/media-fabric.md §3): a finalized VOD queues exactly one
++// object.cmaf (a second finalize joins it); nothing is queued with MEDIA_HLS_ENABLED off, for a non-media object or once
++// a timeline exists; GET …/download?format=json queues it lazily for an older VOD. A playlist token (purpose 'hls',
++// MEDIA_HLS_PLAYLIST_TTL_S, clamped to 12 h) outlives the 1 h download token, is accepted only by the HLS routes and is
++// carried onto the segment URIs; a download signature still opens the HLS routes. format=mp4 queues object.remux once
++// while the remux variant is missing and answers the variant's signed URL once it exists.
++const assert = require('assert');
++const fs = require('fs');
++const os = require('os');
++const path = require('path');
++const http = require('http');
++const express = require('express');
++
++(async () => {
++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-timeline-queue-'));
++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
++ Object.assign(process.env, {
++ VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'), THUMBNAILS_PATH: dir('thumbnails'),
++ PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'), MEDIA_PUBLIC_URL: 'https://media.test',
++ MEDIA_HLS_ENABLED: '1', MEDIA_HLS_PLAYLIST_TTL_S: '999999', MEDIA_INVARIANT_SCAN_HOURS: '0',
++ });
++
++ const config = require('../server/config');
++ assert.strictEqual(config.hls.playlistTtlS, 43200, 'MEDIA_HLS_PLAYLIST_TTL_S is clamped to 12 h');
++ const db = require('../server/db/database');
++ const model = require('../server/objects/model');
++ const timeline = require('../server/objects/timeline');
++ const signing = require('../server/objects/signing');
++ const { queueCmaf } = require('../server/objects/timeline-queue');
++ const tools = require('../server/vod/media-tools');
++ const thumbService = require('../server/thumbnails/thumbnail-service');
++ const vodStorage = require('../server/vod/vod-storage');
++ await vodStorage.tierConfig.init(vodStorage.DEFAULTS);
++ await require('../server/objects/tier-policy').init();
++ const { finalizeVod } = require('../server/vod/finalize');
++ await db.upsertApp({ app_id: 'live', api_key: 'live-key-timeline-queue' });
++
++ // No ffmpeg in these paths; the job worker is never started, so queued jobs stay queued.
++ thumbService.generateVodThumbnail = async () => null;
++ tools.remuxForSeekingDetailed = async () => ({ ok: true, seconds: 30, error: null });
++ tools.probeDuration = async () => ({ ok: true, seconds: 30, format: {}, streams: [], error: null });
++ tools.remuxForSeeking = async () => true;
++ tools.probeVodInfo = async () => ({ duration: 12 });
++
++ const app = express();
++ app.use(express.json());
++ app.use('/api/v1/:app/vods', require('../server/vod/routes'));
++ app.use('/api/v2/:app/objects', require('../server/objects/routes'));
++ app.use('/o', require('../server/objects/routes').publicRouter);
++ const server = http.createServer(app);
++
++ (async () => {
++ await new Promise((r) => server.listen(0, '127.0.0.1', r));
++ const api = `http://127.0.0.1:${server.address().port}`;
++ const auth = { authorization: 'Bearer live-key-timeline-queue' };
++ const call = async (method, p, body) => {
++ const headers = { ...auth };
++ if (body && !(body instanceof FormData)) headers['content-type'] = 'application/json';
++ const r = await fetch(api + p, { method, headers, body: body && !(body instanceof FormData) ? JSON.stringify(body) : body, redirect: 'manual' });
++ return { status: r.status, body: await r.json().catch(() => null) };
++ };
++ const get = async (url) => {
++ const u = new URL(url, api);
++ const r = await fetch(api + u.pathname + u.search, { redirect: 'manual' });
++ return { status: r.status, text: await r.text() };
++ };
++ const jobs = async (type, objectId) => await db.all('SELECT * FROM media_jobs WHERE job_type = ? AND object_id = ?', [type, objectId]);
++ const realNow = Date.now;
++ const at = (s) => { const t0 = realNow(); Date.now = () => t0 + s * 1000; };
++ const media = async (visibility = 'private', extra = {}) => await model.createObject({
++ app_id: 'live', owner_user_id: 5, visibility, lifecycle_status: 'ready', kind: 'file', mime_type: 'video/mp4', size_bytes: 1000, ...extra });
++ const row = (seq, start, end) => ({ seq, name: timeline.segmentName(seq), start_ms: start, end_ms: end, keyframe_ms: seq ? start : null,
++ key: `x/source/${timeline.segmentName(seq)}`, local_path: null, durable_provider: null, packed_object_id: null, byte_offset: null,
++ byte_length: 1000, sha256: 'a'.repeat(64), durability: 'local' });
++
++ // ── Finalize queues the timeline, once ──
++ let r = await call('POST', '/api/v1/live/vods', { title: 'Queued', user_id: 5, visibility: 'private' });
++ assert.strictEqual(r.status, 201);
++ const vodId = r.body.id;
++ const f = new FormData();
++ f.append('chunk', new Blob([Buffer.alloc(40000, 1)], { type: 'video/webm' }), 'a.webm');
++ assert.strictEqual((await call('POST', `/api/v1/live/vods/${vodId}/chunks`, f)).body.status, 'created');
++ const done = await finalizeVod(vodId);
++ assert.ok(done && done.health_status === 'ok' && done.object_id, 'finalize returns the ready row');
++ const vodObj = done.object_id;
++ let cmaf = await jobs('object.cmaf', vodObj);
++ assert.strictEqual(cmaf.length, 1, 'a ready VOD queues one object.cmaf');
++ assert.deepStrictEqual([cmaf[0].status, cmaf[0].created_by, cmaf[0].idempotency_key, cmaf[0].app_id],
++ ['queued', 'system:timeline', `object.cmaf:${vodObj}`, 'live']);
++ await finalizeVod(vodId);
++ assert.strictEqual(await queueCmaf('live', vodObj), cmaf[0].id, 'a later call answers the same job');
++ assert.strictEqual((await jobs('object.cmaf', vodObj)).length, 1, 'a second finalize dedupes');
++ console.log('✅ finalize: one object.cmaf per ready VOD, a second finalize joins it');
++
++ // ── Nothing queued: flag off, non-media, missing, timeline already there ──
++ const plain = await media();
++ config.hls.enabled = false;
++ assert.strictEqual(await queueCmaf('live', plain), null);
++ config.hls.enabled = true;
++ assert.strictEqual((await jobs('object.cmaf', plain)).length, 0, 'flag off: nothing queued');
++ const pdf = await media('private', { mime_type: 'application/pdf' });
++ assert.strictEqual(await queueCmaf('live', pdf), null);
++ assert.strictEqual((await jobs('object.cmaf', pdf)).length, 0, 'a non-media object: nothing queued');
++ assert.strictEqual(await queueCmaf('live', 'med_NOPE'), null, 'a missing object: nothing queued');
++ const cut = await media();
++ await timeline.replace(cut, 'source', [row(0, 0, 0), row(1, 0, 4000), row(2, 4000, 8000)]);
++ assert.strictEqual(await queueCmaf('live', cut), null);
++ assert.strictEqual((await jobs('object.cmaf', cut)).length, 0, 'a timeline already there: nothing queued');
++ console.log('✅ nothing queued with the flag off, for a non-media or missing object, or once the timeline exists');
++
++ // ── Playlist tokens: own purpose, outlive the download token, up to the configured TTL ──
++ const pl = new URL(signing.signedPlaylistUrl(cut).url);
++ assert.strictEqual(pl.pathname, `/o/${cut}/master.m3u8`);
++ const [pexp, psig] = [pl.searchParams.get('exp'), pl.searchParams.get('sig')];
++ const dl = new URL(signing.signedDownloadUrl(cut, 3600).url);
++ const [dexp, dsig] = [dl.searchParams.get('exp'), dl.searchParams.get('sig')];
++ assert.ok(signing.verifyPlaylist(cut, pexp, psig) && signing.verifyDownload(cut, dexp, dsig));
++ assert.ok(!signing.verifyDownload(cut, pexp, psig), 'a playlist token is no download signature');
++ assert.ok(!signing.verifyPlaylist(cut, dexp, dsig), 'a download signature is no playlist token');
++ assert.ok(!signing.verifyPlaylist(plain, pexp, psig), 'bound to its object');
++ at(2 * 3600);
++ assert.ok(signing.verifyPlaylist(cut, pexp, psig), 'valid 2 h on');
++ assert.ok(!signing.verifyDownload(cut, dexp, dsig), 'the 1 h download token is not');
++ at(12 * 3600 - 10);
++ assert.ok(signing.verifyPlaylist(cut, pexp, psig), 'valid up to the configured TTL');
++ at(12 * 3600 + 10);
++ assert.ok(!signing.verifyPlaylist(cut, pexp, psig), 'expired past it');
++ Date.now = realNow;
++ assert.ok(Number(new URL(signing.signedPlaylistUrl(cut, 10 ** 9).url).searchParams.get('exp')) <= Math.floor(realNow() / 1000) + 43200, 'TTL capped at 12 h');
++ console.log('✅ playlist tokens: purpose hls, valid past the download TTL and up to MEDIA_HLS_PLAYLIST_TTL_S');
++
++ // ── The routes: hls_url signed as a playlist, accepted by HLS only, carried onto the segments ──
++ r = await call('GET', `/api/v2/live/objects/${cut}/download?format=json`);
++ assert.strictEqual(r.status, 200, JSON.stringify(r.body));
++ const hls = new URL(r.body.hls_url);
++ assert.strictEqual(hls.pathname, `/o/${cut}/master.m3u8`);
++ assert.ok(signing.verifyPlaylist(cut, hls.searchParams.get('exp'), hls.searchParams.get('sig')), 'hls_url carries a playlist token');
++ assert.ok(!signing.verifyDownload(cut, hls.searchParams.get('exp'), hls.searchParams.get('sig')));
++ assert.ok(Date.parse(r.body.hls_expires_at) - Date.parse(r.body.expires_at) > 3600 * 1000 * 10, 'the playlist outlives the download URL');
++ const durl = new URL(r.body.url);
++ at(2 * 3600);
++ const master = await get(r.body.hls_url);
++ assert.strictEqual(master.status, 200, 'the playlist plays 2 h on');
++ assert.ok(master.text.includes(`source/index.m3u8${hls.search}`), master.text);
++ const index = await get(`/o/${cut}/source/index.m3u8${hls.search}`);
++ assert.strictEqual(index.status, 200);
++ assert.ok(index.text.includes(`000001.m4s${hls.search}`) && index.text.includes(`init.mp4${hls.search}`), 'the token is carried onto the segment URIs');
++ assert.strictEqual((await get(`/o/${cut}/master.m3u8${durl.search}`)).status, 404, 'the download signature has expired by then');
++ Date.now = realNow;
++ assert.strictEqual((await get(`/o/${cut}${hls.search}`)).status, 404, 'GET /o/:id refuses a playlist token');
++ assert.strictEqual((await get(`/o/${cut}/master.m3u8${durl.search}`)).status, 200, 'a download signature still opens the HLS routes');
++ assert.strictEqual((await get(`/o/${cut}/master.m3u8`)).status, 404, 'unsigned: refused');
++ const pubCut = await media('public');
++ await timeline.replace(pubCut, 'source', [row(0, 0, 0), row(1, 0, 4000)]);
++ assert.strictEqual((await call('GET', `/api/v2/live/objects/${pubCut}/download?format=json`)).body.hls_url, `${config.publicUrl}/o/${pubCut}/master.m3u8`, 'public stays unsigned');
++ console.log('✅ hls_url: a playlist token, refused by GET /o/:id, carried onto segments; download signatures still play');
++
++ // ── Lazy: an older VOD gets its timeline queued on the first JSON download ──
++ const old = await media();
++ r = await call('GET', `/api/v2/live/objects/${old}/download?format=json`);
++ assert.strictEqual(r.status, 200);
++ assert.ok(!('hls_url' in r.body), 'no hls_url until the timeline exists');
++ await call('GET', `/api/v2/live/objects/${old}/download?format=json`);
++ cmaf = await jobs('object.cmaf', old);
++ assert.strictEqual(cmaf.length, 1, 'queued once on demand');
++ assert.strictEqual(cmaf[0].created_by, 'system:timeline');
++ assert.strictEqual((await jobs('object.cmaf', pdf)).length, 0);
++ await call('GET', `/api/v2/live/objects/${pdf}/download?format=json`);
++ assert.strictEqual((await jobs('object.cmaf', pdf)).length, 0, 'a non-media download queues nothing');
++ console.log('✅ download?format=json queues object.cmaf lazily, once');
++
++ // ── format=mp4: object.remux once while missing, the variant's URL once present ──
++ r = await call('GET', `/api/v2/live/objects/${plain}/download?format=mp4`);
++ assert.strictEqual(r.status, 202, JSON.stringify(r.body));
++ const job = r.body.job_id;
++ assert.ok(/^mjob_/.test(job));
++ r = await call('GET', `/api/v2/live/objects/${plain}/download?format=mp4`);
++ assert.deepStrictEqual([r.status, r.body.job_id], [202, job], 'joins the queued remux');
++ assert.strictEqual((await jobs('object.remux', plain)).length, 1, 'object.remux queued once');
++ const mp4 = await media('private');
++ await model.setVariant(plain, 'remux', mp4, 'object.remux@1');
++ r = await call('GET', `/api/v2/live/objects/${plain}/download?format=mp4`);
++ assert.strictEqual(r.status, 200, JSON.stringify(r.body));
++ const mu = new URL(r.body.url);
++ assert.deepStrictEqual([mu.pathname, r.body.object_id], [`/o/${mp4}`, mp4]);
++ assert.ok(signing.verifyDownload(mp4, mu.searchParams.get('exp'), mu.searchParams.get('sig')), 'the variant\'s signed URL');
++ assert.strictEqual((await jobs('object.remux', plain)).length, 1, 'nothing more queued');
++ assert.strictEqual((await call('GET', `/api/v2/live/objects/${pdf}/download?format=mp4`)).status, 400, 'no MP4 of a non-media object');
++ console.log('✅ format=mp4: object.remux queued once while missing, the remux variant\'s URL once present');
++
++ server.close();
++ fs.rmSync(tmp, { recursive: true, force: true });
++ console.log('✅ All timeline queue tests passed');
++ process.exit(0);
++ })().catch((err) => { console.error(err); process.exit(1); });
++})().catch((err) => { console.error(err); process.exit(1); });