openvibers/openvibe.media Commits

Index video objects as CMAF segments and serve HLS from the timeline

OP OpenVibers committed 4 Oct 2026 · 214a810a1d · 18 files, +734 −9

Asked: T4 F3.1 — OpenVibe.Media: segment-native video, first slice: the `media_timeline` index plus a CMAF/HLS representation of finished video objects. ONE pull request from current origin/main (git fetch first; main is 0b0ca81: F2 closed out by #24). Design: docs/media-fabric.md §3 "Video is segment-nat…
Harness job media-t4-f3-1-openvibe-media-segment-nat-102734 (plan T4).

Changes

ddiff --git a/.env.example b/.env.example
iindex 0c18279..9ddc9e3 100644
---- a/.env.example
++++ b/.env.example
@@@ -135,3 +135,8 @@ MEDIA_CAPABILITY_GATE=report
#MEDIA_JOBS_HEAVY_WHILE_RECORDING=0
#MEDIA_INVARIANT_SCAN_HOURS=24 # the size-invariant validator (proposes split/remux; never runs them)
#MEDIA_MOVE_CLEANUP_MINUTES=60 # storage.move.cleanup: alert on objects whose placement move failed 3 times (0 = never)
++
++# ── Segment-native video (docs/media-fabric.md §3, F3.1) ─────
++# 1 = the object.cmaf job (CMAF segments of a finished video, indexed in media_timeline) and the HLS routes
++# /o/:id/master.m3u8, /o/:id/source/… ; off (default): the job is refused and the routes 404, nothing else changes.
++#MEDIA_HLS_ENABLED=0
ddiff --git a/README.md b/README.md
iindex a283f8b..4f7427c 100644
---- a/README.md
++++ b/README.md
@@@ -107,6 +107,7 @@ server/
has_result, never params/result/error text/creator), worker (light/heavy lanes, leases, retries,
cancellation), routes (/api/v2/:app/jobs), types: thumbnail.regenerate,
invariant.scan (proposes split/remux), object.split, object.remux, object.waveform, object.sprite,
++ object.cmaf (CMAF/HLS segments of a finished video into media_timeline; MEDIA_HLS_ENABLED),
vod.finalize (orphans and failed finalizes, with backoff),
vod.duration.reconcile (stored vs measured durations, local and B2/R2),
storage.orphans.scan (monthly storage orphan report, service-wide, report only)
@@@ -456,6 +457,7 @@ Counters are per process (a restart forgets them). `test/actor-limits.test.js`.
| `GET /live/:sel/transcript.json` | **transcript + AI timeline API** — full audio-transcription log and AI overview timeline. `:sel` = slot id / slot slug → slot-scoped sessions; **`@username`** → user-scoped (all their slots, works while offline). Returns `{ live, current, sessions[] (each: ai_overview, transcript, duration…), streamer (overview + stream memories), user }`. `?limit=1..50` sessions (default 10), `?app=`. Cached **30s** (that's the rate limit), CORS-open. |
| `GET /live/:sel/chat-insight.json` | **chat insight API** — a user's chat-related AI insight/timeline (`:sel` = `@username` or numeric user id): today-vs-alltime chat overviews, condensed memory, event timeline, plus their streamer overview + stream memories when they stream. Proxied from the app's public chat-AI API over loopback; cached **30s**, CORS-open. |
| `GET /v/:id/transcript.json` | **VOD transcript API** — transcript + AI overview for one existing VOD id (`{ vod_id, title, duration_seconds, ai_overview, transcript, ai_analyzed_at }`). Private VODs → 404. Cached 30s, CORS-open. |
++| `GET /o/:id/master.m3u8`, `/o/:id/source/index.m3u8`, `/o/:id/source/{init.mp4,NNNNNN.m4s}` | segment-native video (F3.1), only with `MEDIA_HLS_ENABLED=1` (off by default; off, these 404): HLS playlists written from the object's `media_timeline` rows (VOD, `#EXT-X-ENDLIST`) and its CMAF segments (Range). Same check as `GET /o/:id`: private and sandbox objects only with the object's `?exp&sig` (`/api/v2/:app/objects/:id/download?format=json` → `hls_url`), carried onto every URI; deleted 410. The `object.cmaf` job makes the timeline. `noindex` |
| `GET /robots.txt` | crawler policy (`openvibe-shared/seo.robotsTxt`: AI and search crawlers named, `/api/`, `/auth/`, `/internal/`, `/o/`, `/metrics`, `/live/`, `/me` disallowed) and the sitemap |
| `GET /sitemap.xml` | the media index and the watch pages Media is the canonical home of: public, playable VODs and clips of apps other than Live (Live lists its own `/vod` and `/clip` pages), never AI clips, private, unlisted, sandbox, recording or failed items; thumbnails as image entries. Every listed page renders `index, follow` |
| `GET /llms.txt` | a map of the site for language-model crawlers |
ddiff --git a/STATUS.json b/STATUS.json
iindex c787584..813ab9b 100644
---- a/STATUS.json
++++ b/STATUS.json
@@@ -36,7 +36,8 @@
"per-actor limits (openvibe-sdk/limits)",
"placement demand rollups on Valkey (F2.4): reads per object × region × 5-minute bucket, hotness over the hour, media.object.hot once per object per hour",
"tiering on demand (F2.5): the object sweep promotes and demotes on the hour's Valkey reads against per-class hysteresis bands, one MGET per page, falling back to the daily PostgreSQL counts for a whole sweep when Valkey fails",
-- "placement constraints (F2.6): sweep moves refused while their provider fails its class (breaker or probe), and a per-class monthly R2 storage ceiling (maxHotUsdPerMonth) applied in value order, each explained in the decision's inputs"
++ "placement constraints (F2.6): sweep moves refused while their provider fails its class (breaker or probe), and a per-class monthly R2 storage ceiling (maxHotUsdPerMonth) applied in value order, each explained in the decision's inputs",
++ "segment-native video (F3.1, behind MEDIA_HLS_ENABLED): the media_timeline index, the object.cmaf job (CMAF/fMP4 segments of a finished video on its own keyframes, sha256 and durability per row, idempotent) and HLS playlists written from the timeline at /o/:id/master.m3u8 under the object's access checks"
],
"updated": "2026-10-03"
}
ddiff --git a/docs/media-fabric.md b/docs/media-fabric.md
iindex 9e255e4..d6a8a17 100644
---- a/docs/media-fabric.md
++++ b/docs/media-fabric.md
@@@ -87,6 +87,31 @@ med_xyz/{captions/*.vtt, storyboard.webp+.vtt, waveform, metadata}
the shared axis for DVR, seek, clips, sprites, captions, transcripts, chapters, AI moments and moderation markers.
- **MP4 stays for downloads and old clients:** a remuxed faststart MP4 (lazy, cached like any derivative) with plain
HTTP ranges.
++- **F3.1, shipped: the timeline index and the source representation of finished video.** Migration
++ `0002_media_timeline.sql` adds `media_timeline` (object, rendition, segment number `seq` — 0 is the init segment —,
++ `start_ms`/`end_ms` (contiguous, half-open), `keyframe_ms`, the segment's `key` `<object>/<rendition>/<name>`,
++ `local_path`, `durable_provider`, `packed_object_id` + `byte_offset`/`byte_length` (packing is F3.3; NULL until then),
++ `sha256`, `durability` `local|durable`), keyed by (object, rendition, seq) for "segments in order" with an index on
++ (object, rendition, start_ms) for "the segment at t". `server/objects/timeline.js` is the store (`segmentAt(objectId,
++ rendition, tMs)`, `segments`, `replace`, `removeObject`, the playlist writers). The heavy-lane job **`object.cmaf`**
++ (`server/jobs/cmaf.js`) stream-copies a ready vod/clip/video object (`ffmpeg -c copy -f hls -hls_segment_type fmp4`,
++ `params.segment_seconds` 1–10, 4 by default, cut at the first source keyframe after it) into `init.mp4` +
++ `000001.m4s`…, stores each under `OBJECTS_PATH/.timeline/<app>/<object>/source/` and, with B2 configured, under the
++ key `<object>/source/<name>` on B2 (`durable` once B2 confirms the size), then writes all rows of the rendition in one
++ transaction. It never touches the source object; a rerun leaves rows whose sha256 matches as they are and uploads
++ only what is not durable yet; it honours the abort signal and the stream-copy budget and removes its work directory.
++ Behind **`MEDIA_HLS_ENABLED`** (off by default: the job is refused with `409 media.hls.disabled` and the routes 404),
++ `GET /o/:id/master.m3u8`, `/o/:id/source/index.m3u8` and `/o/:id/source/{init.mp4,NNNNNN.m4s}` serve the timeline
++ under the same check as `GET /o/:id` (private and sandbox objects only with the object's `?exp&sig`, carried onto
++ every URI of a signed playlist; deleted 410). The playlists are written from the rows (`#EXT-X-PLAYLIST-TYPE:VOD`,
++ `#EXT-X-ENDLIST`, EXTINF = the row's duration, BANDWIDTH = peak segment bitrate); `GET …/download?format=json` adds
++ `hls_url` when a timeline exists. A purge, or a vod/clip deleted for good, removes the segments and rows; a held
++ object keeps them. Storage orphan reports count timeline files and keys as wanted.
++ **Still open in F3:** the growing live/DVR playlist written as OpenRe segments (F3.2), write-behind durability for
++ live segments and its upload-lag metric, packing into ~60 s chunks (F3.3), virtual and materialized clips over the
++ timeline, sprites and captions on it, the faststart MP4 fallback as a derivative, segment-bucket demand, a finalize
++ hook that queues `object.cmaf` by itself, signed playlists that outlive one signed-URL lifetime, and Live's player
++ moving to HLS. Renditions are F4.
## 4. Clips reuse the source
@@@ -137,7 +162,7 @@ formats (AVIF/WebP), preview clips and AI thumbnails are `rebuildable = true`:
`<bucket>` is the 5-minute bucket number (`floor(epoch_ms / 300000)`). The region is one per deployment
(`MEDIA_DEMAND_REGION`, default `local`); per-viewer regions wait for an edge-provided header. `hotness()` sums the last
12 buckets (one hour) per object. The writes are fire-and-forget: a slow, failing or absent Valkey never delays or fails
-- a read (without Valkey the counts stay in-process). Object × region only for now; the segment bucket comes with F3.
++ a read (without Valkey the counts stay in-process). Object × region only for now; the segment bucket comes later in F3 (F3.1 shipped the timeline it will bucket).
- **The sweep reads demand (F2.5, shipped):** the object tiering sweep's promote/demote eligibility is `hotness()` for the
deployment's region, against each class's hysteresis band in `media.storage_policy` (`promoteReadsPerHour` /
`demoteReadsPerHour`, demote < promote, validated per revision): video 60 / 6, image 300 / 30, download 30 / 3, the
@@@ -237,8 +262,9 @@ metric. Budgets per class and provider with a forecast; `media.provider.cost_thr
eligibility on them with per-class hysteresis and a PostgreSQL fallback (F2.5, shipped), the provider-class gate
and per-class monthly R2 storage ceilings on its moves (F2.6, shipped), the move cleanup job and its alert after
three failures (§6, shipped). Still open in F2: every class (not only native objects ↔ R2) through the one sweep.
--- **F3 segment-native video:** CMAF/HLS recording with the growing DVR playlist, the timeline index, packing after
-- finalize, virtual clips, sprites and captions on the timeline, MP4 fallback; Live's player moves to HLS.
++- **F3 segment-native video:** the timeline index and the CMAF/HLS source representation of finished video (F3.1,
++ shipped; §3); still open: CMAF/HLS recording with the growing DVR playlist from OpenRe, packing after finalize,
++ virtual clips, sprites and captions on the timeline, MP4 fallback; Live's player moves to HLS.
- **F4 reactive derivatives:** on-demand renditions and image variants, keep-vs-regenerate economics, AV1 for viral
VODs, compute placement.
- **F5 multi-CDN delivery:** Bunny path (B2 origin), R2 custom-domain path, OpenVibe edge first up to capacity, route
ddiff --git a/docs/object-model.md b/docs/object-model.md
iindex 6cc4211..48fcd54 100644
---- a/docs/object-model.md
++++ b/docs/object-model.md
@@@ -27,6 +27,7 @@ Code: `server/objects/` (`model.js`, `routes.js`, `backfill.js`, `reconcile.js`,
| `media_holds` | retention holds (with a staff `note`), see [Holds](#retention-holds) |
| `media_namespaces` | every namespace: `namespace`, `app_id` (tenant), `parent`, `owner` (`service:<app>` or `project:<prj_…>`), `policy` (JSON), `quota_bytes`, `quota_objects`, and the usage snapshot `used_bytes`, `used_objects`, `reserved_bytes`, `reserved_objects`, `reconciled_at`. See [Namespaces](#namespaces-grants-and-quotas) |
| `media_quota_reservations` | quota held by an upload in progress: `object_id`, `app_id`, `namespace`, `bytes`, `expires_at` |
++| `media_timeline` | the timeline index of segment-native video (F3.1, [media-fabric §3](media-fabric.md#3-video-is-segment-native-cmaf-under-hls)): one row per CMAF segment of one rendition (`object_id`, `rendition`, `seq`: 0 is `init.mp4`), `start_ms`/`end_ms`/`keyframe_ms`, `key` (`<object>/<rendition>/<name>`), `local_path`, `durable_provider`, `packed_object_id` + `byte_offset`/`byte_length`, `sha256`, `durability` (`local` `durable`). Written by `object.cmaf`; read by the HLS routes and `timeline.segmentAt` |
| `media_invariant_violations` | public playback objects over the size policy, see [Invariant](#public-object-size-invariant) |
Location `state`:
@@@ -758,6 +759,7 @@ job over only with the token it read and, past boot, only while the lease is sti
| `object.hash` | light | sha256 of ready objects' present local copies that have no hash yet, onto the object (`content_hash`, `metadata.hash_basis`) and the local location (`checksum`). Tenant-wide batch (`params { limit (1-500, 50), budget_mb }`) or one object. Scheduled per tenant every `MEDIA_HASH_INTERVAL_MIN`. Result `{ hashed, bytes, skipped, remaining }`. See [Scheduled verification](#scheduled-verification) |
| `vod.duration.reconcile` | heavy | One bounded batch of the tenant's finished VODs: stored duration vs a measurement of the real file, local or the B2/R2 copy (ffprobe over a presigned URL, ranged reads). `params { limit (1-200, 25), after_id?, apply (false), confirm_remote (false) }`; without `after_id` it walks the library from where the last run stopped. Repairs only confirmed, clearly wrong values when `apply`. Result `{ counts, range, next_after_id, report }`. See [Duration reconciliation](#duration-reconciliation) |
| `storage.orphans.scan` | light | The [storage orphan report](#storage-orphan-report), service-wide, under the system app `_media`; report only. `params { limit (1-100000, 2000) }`. Result `{ totals, providers, summary, report }` (the report file under `<data>/reports/`). Scheduled every `MEDIA_ORPHAN_SCAN_DAYS`; tenants cannot enqueue it (403) |
++| `object.cmaf` | heavy | Segment-native video (F3.1; only with `MEDIA_HLS_ENABLED`, else `409 media.hls.disabled`): a stream copy of a ready vod/clip/video object into CMAF segments on its own keyframes (`init.mp4` + `NNNNNN.m4s`; `segment_seconds` 1–10, default 4: a segment ends at the first keyframe after it), stored under `OBJECTS_PATH/.timeline/<app>/<object>/source/` and, with B2 configured, as `<object>/source/<name>` on B2; one `media_timeline` row per segment (times, size, sha256, `durability`), all written in one transaction. The source is untouched. A rerun keeps rows whose sha256 matches and uploads only what is not durable; a failed upload leaves its row `local` and retries the job. Result `{ source_id, rendition, segments, duration_ms, bytes, durable, local_only, rows: { inserted, updated, unchanged, removed }, uploaded, master_path }`. Served at `/o/:id/master.m3u8` (see [media-fabric §3](media-fabric.md#3-video-is-segment-native-cmaf-under-hls)) |
| `storage.move.cleanup` | light | Placement moves whose delete keeps failing: an object with three `failed` rows in `media_object_tier_decisions` in the last 7 days (a later `done` resets them) raises one `storage.alert` of kind `move_cleanup_failed`. Service-wide under the system app `_media`; report only. `params {}`. Result `{ scanned, alerted, objects }`. Scheduled every `MEDIA_MOVE_CLEANUP_MINUTES` (60; `0` = never); tenants cannot enqueue it (403) |
| `vod.finalize` | finalize | Finalizes a recording whose finalize failed or never ran. `params { vod_id }`. Queued by the orphan sweep (a row still `is_recording = 1` that no recorder, chunk upload or finalize holds and whose file has been idle for `MEDIA_FINALIZE_ORPHAN_GRACE_S`, 180 s; at boot and every `MEDIA_FINALIZE_SWEEP_S`, 120 s) and by finalize itself when it could not settle a recording (nothing measurable, a stat failure, a throw; first run after 5 min). Still unmeasurable: retried (5 min, 15 min, 45 min, 2 h 15, 6 h; 6 attempts), then the VOD stays `needs_review`, hidden. Never stops a live recording. Result `{ outcome: ready \| needs_review \| corrupt \| deleted \| skipped, duration_seconds, duration_source }` |
@@@ -984,6 +986,7 @@ deleted object's R2 copy stays (so its purge waits) until the gate is on again.
| `MEDIA_UPLOAD_MIN_FREE_MB` | 10240 | free disk kept in reserve by multipart uploads and split/remux jobs |
| `MEDIA_UPLOAD_RESERVATION_HOURS` | 72 | an upload's quota reservation lasts this long after its last step; then the hourly sweep fails it and frees its bytes |
| `MEDIA_NAMESPACE_MAX_CHILDREN` | 100 | namespaces a tenant may have below its root |
++| `MEDIA_HLS_ENABLED` | off | `1` turns segment-native video on: the `object.cmaf` job and the HLS routes `/o/:id/master.m3u8`, `/o/:id/source/…` (off: the job is refused, the routes 404) |
| `MEDIA_JOBS_ENABLED` | on | `0` stops the job worker |
| `MEDIA_JOBS_POLL_MS` | 5000 | worker poll interval |
| `MEDIA_JOBS_LIGHT_CONCURRENCY` / `_HEAVY_CONCURRENCY` / `_FINALIZE_CONCURRENCY` | 2 / 1 / 1 | jobs per lane |
ddiff --git a/migrations/0002_media_timeline.sql b/migrations/0002_media_timeline.sql
nnew file mode 100644
iindex 0000000..1eda5a3
---- /dev/null
++++ b/migrations/0002_media_timeline.sql
@@@ -0,0 +1,34 @@
++-- phase: expand
++-- Segment-native video, first slice (docs/media-fabric.md §3, F3.1): the timeline index. One row per CMAF segment of one
++-- rendition of a media object; segment 0 is the rendition's init segment (init.mp4, start = end = 0). Times are integer
++-- milliseconds on the object's own timeline: [start_ms, end_ms) is contiguous from one segment to the next and
++-- keyframe_ms is the segment's first keyframe. The bytes are the segment's own key under the object's prefix
++-- (<object>/<rendition>/<name>) on this node's disk (local_path) and, once durable, on durable_provider; packed_object_id
++-- with byte_offset/byte_length names the ~60 s chunk object a packed segment lives in (F3.3; NULL until then).
++-- durability: 'local' = only on this node's disk, 'durable' = the durable provider confirmed the bytes.
++-- Additive only: an older release never reads it. Never edited after it runs; a further change is a new migration.
++CREATE TABLE media_timeline (
++ object_id text COLLATE "C" NOT NULL,
++ rendition text COLLATE "C" NOT NULL,
++ seq bigint NOT NULL CHECK (seq >= 0),
++ name text COLLATE "C" NOT NULL,
++ start_ms bigint NOT NULL,
++ end_ms bigint NOT NULL,
++ keyframe_ms bigint,
++ key text COLLATE "C" NOT NULL,
++ local_path text COLLATE "C",
++ durable_provider text COLLATE "C" CHECK (durable_provider IN ('b2', 'r2')),
++ packed_object_id text COLLATE "C",
++ byte_offset bigint,
++ byte_length bigint NOT NULL,
++ sha256 text COLLATE "C" NOT NULL,
++ durability text COLLATE "C" NOT NULL DEFAULT 'local' CHECK (durability IN ('local', 'durable')),
++ job_id text COLLATE "C",
++ created_at text COLLATE "C" DEFAULT ov_now(),
++ updated_at text COLLATE "C" DEFAULT ov_now(),
++ PRIMARY KEY (object_id, rendition, seq),
++ CHECK (end_ms >= start_ms)
++);
++-- "segments of a rendition in order" is the primary key; "the segment at time t" walks this one backwards from t.
++CREATE INDEX media_timeline_at ON media_timeline (object_id, rendition, start_ms);
++CREATE INDEX media_timeline_packed ON media_timeline (packed_object_id) WHERE packed_object_id IS NOT NULL;
ddiff --git a/server/config.js b/server/config.js
iindex 7805e53..9c438c5 100644
---- a/server/config.js
++++ b/server/config.js
@@@ -85,6 +85,12 @@ const config = {
},
// Job system (server/jobs/; docs/object-model.md#jobs).
++ // Segment-native video (docs/media-fabric.md §3, F3.1): the object.cmaf job and the HLS playlists read from
++ // media_timeline (/o/:id/master.m3u8, /o/:id/source/…). Off by default: off, the job is refused and the routes 404.
++ hls: {
++ enabled: ['1', 'true', 'on'].includes(String(process.env.MEDIA_HLS_ENABLED || '').toLowerCase()),
++ },
++
jobs: {
enabled: !['0', 'false', 'off'].includes(String(process.env.MEDIA_JOBS_ENABLED || '').toLowerCase()),
pollMs: intEnv('MEDIA_JOBS_POLL_MS', 5000),
ddiff --git a/server/jobs/cmaf.js b/server/jobs/cmaf.js
nnew file mode 100644
iindex 0000000..4325d25
---- /dev/null
++++ b/server/jobs/cmaf.js
@@@ -0,0 +1,172 @@
++/**
++ * OpenVibe.Media — job type object.cmaf (heavy lane; docs/media-fabric.md §3, F3.1)
++ *
++ * The source representation of a finished video (or audio) object as CMAF under HLS: a stream copy (ffmpeg -c copy,
++ * no re-encode) into fragmented-MP4 segments cut on the source's own keyframes (variable duration), init.mp4 plus
++ * 000001.m4s, …, stored under the object's `source/` prefix and indexed in media_timeline (objects/timeline.js), one
++ * row per segment with its times, size, sha256 and durability. The source object is never touched: its bytes,
++ * locations, visibility, metadata and lifecycle stay as they are.
++ *
++ * params { segment_seconds } the target segment length, 1-10 (4): a segment ends at the first keyframe after it
++ * result { source_id, rendition: 'source', segments, duration_ms, bytes, durable, local_only,
++ * rows: { inserted, updated, unchanged, removed }, uploaded, master_path }
++ *
++ * Bytes: each segment lands on this node's disk (OBJECTS_PATH/.timeline/…); with B2 configured it is uploaded under
++ * <object>/source/<name> and its row says `durable` once B2 confirms the size. Idempotent: a rerun makes the same
++ * bytes (bitexact), leaves a row whose sha256 matches as it is, and uploads only what is not durable yet. The rows of
++ * a rendition are written in one transaction after every segment is stored, so a half-finished run leaves no
++ * timeline. An upload that fails leaves its row `local` and fails the attempt, so the retry finishes it.
++ * Refused: MEDIA_HLS_ENABLED off, a source that is not ready or not media (permanent), less free disk than the source
++ * size plus MEDIA_UPLOAD_MIN_FREE_MB (retried later), a codec fMP4 cannot carry (permanent).
++ */
++'use strict';
++
++const fs = require('fs');
++const path = require('path');
++const config = require('../config');
++const timeline = require('../objects/timeline');
++const derive = require('./derive');
++const { JobError } = require('./queue');
++
++const MB = 1024 * 1024;
++const DEFAULT_SEGMENT_SECONDS = 4;
++const DURABLE = 'b2';
++
++function disabled() {
++ return new JobError('media.hls.disabled', 'Segment-native video is off on this server (MEDIA_HLS_ENABLED)', { permanent: true, status: 409 });
++}
++
++function validate({ obj, params }) {
++ if (!config.hls.enabled) throw disabled();
++ if (!obj) throw new JobError('media.job.invalid', 'object.cmaf needs object_id', { permanent: true });
++ if (!derive.isMediaObject(obj)) throw new JobError('media.job.invalid', 'Only vod/clip objects (or video/audio objects) have a timeline', { permanent: true });
++ const p = { ...(params || {}) };
++ if (p.segment_seconds != null) {
++ const s = Number(p.segment_seconds);
++ if (!Number.isFinite(s) || s < 1 || s > 10) throw new JobError('media.job.invalid', 'params.segment_seconds must be from 1 to 10', { permanent: true });
++ p.segment_seconds = s;
++ }
++ return p;
++}
++
++/** The segments ffmpeg wrote, in order: [{ name, seconds }] from its playlist (#EXTINF before each URI). */
++function parsePlaylist(text) {
++ const out = [];
++ let dur = null;
++ for (const line of String(text).split(/\r?\n/)) {
++ const inf = /^#EXTINF:([\d.]+)/.exec(line);
++ if (inf) { dur = Number(inf[1]); continue; }
++ if (line && !line.startsWith('#') && dur != null) { out.push({ name: line.trim(), seconds: dur }); dur = null; }
++ }
++ return out;
++}
++
++function checkDisk(needBytes) {
++ const reserve = config.objects.uploadMinFreeMb * MB;
++ let free = Infinity;
++ try { const st = fs.statfsSync(config.objects.path); free = Number(st.bavail) * Number(st.bsize); } catch { /* unknown */ }
++ if (free < needBytes + reserve) {
++ throw new JobError('insufficient_disk', `Not enough free disk: need ${Math.ceil((needBytes + reserve) / MB)} MB, ${Math.floor(free / MB)} MB free`, { retryAfterS: 1800 });
++ }
++}
++
++const aborted = (ctx) => { if (ctx.signal.aborted) throw ctx.signal.reason || new Error('aborted'); };
++
++async function run(job, ctx) {
++ if (!config.hls.enabled) throw disabled();
++ const src = await derive.loadSource(job);
++ const source = await derive.resolveSource(src);
++ if (!source) throw new JobError('media_unavailable', 'The source bytes are unavailable (no local file and no cloud copy)', { permanent: true });
++ checkDisk(Number(src.size_bytes) || 0);
++ const rendition = timeline.SOURCE;
++ const dir = derive.workDir(job.id);
++ try {
++ const out = path.join(dir, 'out');
++ fs.mkdirSync(out, { recursive: true });
++ const r = await derive.ffmpeg(['-y', '-nostdin', '-v', 'error', ...derive.inputArgs(source), '-i', source.input,
++ '-map', '0:v:0?', '-map', '0:a:0?', '-c', 'copy', '-map_metadata', '-1', '-fflags', '+bitexact', '-flags', '+bitexact',
++ '-f', 'hls', '-hls_time', String(job.params.segment_seconds || DEFAULT_SEGMENT_SECONDS), '-hls_list_size', '0',
++ '-hls_playlist_type', 'vod', '-hls_segment_type', 'fmp4', '-hls_fmp4_init_filename', timeline.INIT_NAME,
++ '-start_number', '1', '-hls_segment_filename', path.join(out, '%06d.m4s'), path.join(out, 'index.m3u8')],
++ { signal: ctx.signal, timeoutMs: derive.budgetMs(src.size_bytes) });
++ aborted(ctx);
++ const tail = String(r.stderr).trim().split('\n').slice(-2).join(' ').slice(0, 300);
++ if (!r.ok) {
++ const unsupported = /not currently supported in container|could not find tag for codec|codec not supported/i.test(r.stderr);
++ throw new JobError(unsupported ? 'codec_unsupported' : 'ffmpeg_failed', `ffmpeg exited ${r.code}: ${tail}`, { permanent: unsupported });
++ }
++ const list = derive.existing(path.join(out, 'index.m3u8')) ? parsePlaylist(fs.readFileSync(path.join(out, 'index.m3u8'), 'utf8')) : [];
++ if (!list.length || !derive.existing(path.join(out, timeline.INIT_NAME))) throw new JobError('ffmpeg_failed', `ffmpeg wrote no segments: ${tail}`);
++
++ // Times on the object's own axis: cumulative EXTINF in ms, so each segment ends where the next starts.
++ const files = [{ seq: 0, name: timeline.INIT_NAME, start_ms: 0, end_ms: 0, keyframe_ms: null }];
++ let cum = 0;
++ list.forEach((s, i) => {
++ const start = Math.round(cum * 1000);
++ cum += s.seconds;
++ files.push({ seq: i + 1, name: timeline.segmentName(i + 1), file: s.name, start_ms: start, end_ms: Math.round(cum * 1000), keyframe_ms: start });
++ });
++
++ const prev = new Map((await timeline.list(src.id, rendition)).map((x) => [Number(x.seq), x]));
++ const vodStorage = require('../vod/vod-storage');
++ const durableOn = vodStorage.providerConfigured(DURABLE);
++ const rows = [];
++ let uploaded = 0;
++ const failed = [];
++ for (const f of files) {
++ aborted(ctx);
++ const tmp = path.join(out, f.file || f.name);
++ if (!derive.existing(tmp)) throw new JobError('ffmpeg_failed', `ffmpeg listed ${f.file || f.name} but did not write it`);
++ const size = fs.statSync(tmp).size;
++ const sha = await derive.sha256File(tmp);
++ const old = prev.get(f.seq);
++ const match = !!old && old.sha256 === sha;
++ const dest = timeline.localPathFor(src, rendition, f.name);
++ if (!(match && derive.existing(dest))) {
++ fs.mkdirSync(path.dirname(dest), { recursive: true });
++ fs.renameSync(tmp, dest);
++ }
++ const key = timeline.keyFor(src.id, rendition, f.name);
++ let durable = match && old.durability === 'durable' ? old.durable_provider : null;
++ if (!durable && durableOn) {
++ try {
++ if (!vodStorage.providerAvailable(DURABLE)) throw new Error(`${DURABLE} is unavailable`);
++ await vodStorage.uploadFile(DURABLE, key, dest, f.seq === 0 ? 'video/mp4' : 'video/iso.segment');
++ durable = DURABLE;
++ uploaded++;
++ } catch (err) { failed.push(`${f.name}: ${err.message}`); }
++ }
++ rows.push({
++ seq: f.seq, name: f.name, start_ms: f.start_ms, end_ms: f.end_ms, keyframe_ms: f.keyframe_ms, key, local_path: dest,
++ durable_provider: durable, packed_object_id: null, byte_offset: null, byte_length: size, sha256: sha,
++ durability: durable ? 'durable' : 'local',
++ });
++ }
++ aborted(ctx);
++ // Segments past the new end (an older run cut the source differently) go with their rows.
++ const last = files[files.length - 1].seq;
++ for (const [seq, old] of prev) {
++ if (seq <= last) continue;
++ if (old.local_path && path.resolve(old.local_path).startsWith(timeline.localRoot() + path.sep)) { try { fs.unlinkSync(old.local_path); } catch { /* gone */ } }
++ if (old.durable_provider && vodStorage.providerConfigured(old.durable_provider)) await vodStorage.deleteObject(old.durable_provider, old.key).catch(() => {});
++ }
++ const written = await timeline.replace(src.id, rendition, rows, { jobId: job.id });
++ if (failed.length) {
++ throw new JobError('upload_failed', `${failed.length} segment(s) not durable yet (${failed[0].slice(0, 200)}); a retry uploads only those`, { retryAfterS: 600 });
++ }
++ const segs = rows.filter((x) => x.seq > 0);
++ return {
++ source_id: src.id, rendition, segments: segs.length, duration_ms: segs[segs.length - 1].end_ms,
++ bytes: rows.reduce((a, x) => a + x.byte_length, 0),
++ durable: rows.filter((x) => x.durability === 'durable').length, local_only: rows.filter((x) => x.durability === 'local').length,
++ rows: written, uploaded, master_path: `/o/${src.id}/master.m3u8`, source_unchanged: true,
++ };
++ } finally {
++ derive.cleanupWork(job.id);
++ }
++}
++
++module.exports = {
++ spec: { lane: 'heavy', maxAttempts: 3, timeoutMs: 3 * 3600 * 1000, needsObject: true, validate, run },
++ parsePlaylist,
++};
ddiff --git a/server/jobs/derive.js b/server/jobs/derive.js
iindex d6cfaa9..fae51bd 100644
---- a/server/jobs/derive.js
++++ b/server/jobs/derive.js
@@@ -316,4 +316,6 @@ module.exports = {
resolveSource, isMediaObject, extFor,
// for the preview jobs (previews.js)
loadSource, adopt, ffmpeg, inputArgs, probeDuration, workDir, cleanupWork, existing, checkRoom,
++ // and object.cmaf (cmaf.js)
++ budgetMs, sha256File,
};
ddiff --git a/server/jobs/types.js b/server/jobs/types.js
iindex 3804e26..354aed8 100644
---- a/server/jobs/types.js
++++ b/server/jobs/types.js
@@@ -7,6 +7,7 @@
* object.remux heavy a stream-copy remux of a vod/clip as a new private object
* object.waveform heavy the audio waveform (PNG) of a vod/clip as the source's `waveform` variant
* object.sprite heavy a seek-preview sprite sheet (JPEG, layout in its metadata) as the `sprite` variant
++ * object.cmaf heavy the source as CMAF/HLS segments on its own keyframes, indexed in media_timeline (MEDIA_HLS_ENABLED)
* vod.finalize finalize finalize a recording whose finalize failed or never ran (orphans), with backoff
* clip.cut clips cut (or re-cut) a clip row from its VOD; retries with backoff (server/vod/clip-jobs.js)
* vod.duration.reconcile heavy stored VOD durations vs a measurement of the real file (local or B2/R2), one batch
@@@ -24,6 +25,7 @@ queue.register('object.split', require('./derive').split);
queue.register('object.remux', require('./derive').remux);
queue.register('object.waveform', require('./previews').waveform);
queue.register('object.sprite', require('./previews').sprite);
++queue.register('object.cmaf', require('./cmaf').spec);
queue.register('vod.finalize', require('./vod-finalize').spec);
queue.register('clip.cut', require('../vod/clip-jobs').spec);
queue.register('vod.duration.reconcile', require('./duration-reconcile').spec);
ddiff --git a/server/jobs/worker.js b/server/jobs/worker.js
iindex 77d58a0..f9334db 100644
---- a/server/jobs/worker.js
++++ b/server/jobs/worker.js
@@@ -3,7 +3,7 @@
*
* Runs media_jobs in-process, in two lanes so a long split never holds up a thumbnail:
* light thumbnail.regenerate, invariant.scan MEDIA_JOBS_LIGHT_CONCURRENCY (2)
-- * heavy object.split, object.remux MEDIA_JOBS_HEAVY_CONCURRENCY (1); waits while a
++ * heavy object.split, object.remux, object.cmaf MEDIA_JOBS_HEAVY_CONCURRENCY (1); waits while a
* recording runs unless MEDIA_JOBS_HEAVY_WHILE_RECORDING=1
* finalize vod.finalize MEDIA_JOBS_FINALIZE_CONCURRENCY (1); runs while recording
* (it is the work the recorder does when a stream ends)
ddiff --git a/server/objects/model.js b/server/objects/model.js
iindex 620ec72..b2a2959 100644
---- a/server/objects/model.js
++++ b/server/objects/model.js
@@@ -675,6 +675,7 @@ async function purgeExpired({ retentionDays = config.objects.retentionDays } = {
if (l.provider === 'local' && path.resolve(l.key).startsWith(root)) { try { fs.unlinkSync(l.key); } catch { /* already gone */ } }
if (l.provider === 'local') await db.run('DELETE FROM media_locations WHERE id = ?', [l.id]);
}
++ await require('./timeline').removeObject(obj.id); // its CMAF segments (F3.1) go with the bytes
md.purged_at = new Date().toISOString();
await db.run('UPDATE media_objects SET metadata = ?, updated_at = ov_now() WHERE id = ?', [JSON.stringify(md), obj.id]);
purged++;
ddiff --git a/server/objects/routes.js b/server/objects/routes.js
iindex a8718e9..09d3dea 100644
---- a/server/objects/routes.js
++++ b/server/objects/routes.js
@@@ -35,7 +35,8 @@
* URL), and the multipart routes the session token, so a browser can send the bytes directly.
*
* Public bytes: GET /o/:id (publicRouter) — public/unlisted objects openly,
-- * private ones only with a valid ?exp&sig from /download.
++ * private ones only with a valid ?exp&sig from /download. With MEDIA_HLS_ENABLED, the object's timeline as HLS
++ * under the same check: /o/:id/master.m3u8, /o/:id/source/index.m3u8, /o/:id/source/{init.mp4,NNNNNN.m4s}.
*/
'use strict';
@@@ -667,13 +668,16 @@ router.get('/:id/download', read, limits('media.object.download'), async (req, r
const json = req.query.format === 'json';
res.set('Cache-Control', cache.htmlHeaders({ private: true }));
// Developer-project sandbox objects are never public, whatever their visibility: always signed.
++ // With segment-native video on (MEDIA_HLS_ENABLED) and a timeline made, the JSON answer also names the HLS master
++ // playlist: open for a public object, under the same signature as `url` for a private one.
++ const hls = json && config.hls.enabled && await require('./timeline').has(obj.id);
if (obj.visibility !== 'private' && !await db.isSandboxTenant(obj.app_id)) {
const url = model.legacyPublicUrl(obj) || `${config.publicUrl}/o/${obj.id}`;
-- return json ? res.json({ url, expires_at: null, public: true }) : res.redirect(302, url);
++ return json ? res.json({ url, expires_at: null, public: true, ...(hls && { hls_url: `${config.publicUrl}/o/${obj.id}/master.m3u8` }) }) : res.redirect(302, url);
}
const signed = signing.signedDownloadUrl(obj.id, req.query.ttl);
if (['1', 'true'].includes(String(req.query.redirect || ''))) return res.redirect(302, signed.url);
-- res.json({ url: signed.url, expires_at: signed.expires_at, public: false });
++ res.json({ url: signed.url, expires_at: signed.expires_at, public: false, ...(hls && { hls_url: signed.url.replace('?', '/master.m3u8?') }) });
});
// ── Retention holds ──────────────────────────────────────────
@@@ -727,6 +731,76 @@ router.delete('/:id/holds/:holdId', appOnly, async (req, res) => {
const INLINE = /^(image\/(?!svg)|video\/|audio\/|application\/pdf$|text\/plain$)/;
const publicRouter = express.Router();
++
++// ── Segment-native video (docs/media-fabric.md §3, F3.1; MEDIA_HLS_ENABLED) ──
++// GET /o/:id/master.m3u8, /o/:id/source/index.m3u8 and /o/:id/source/{init.mp4,NNNNNN.m4s}: the playlists are written
++// from the object's media_timeline rows (objects/timeline.js), never read from files. Every one passes the same check
++// as GET /o/:id: public and unlisted objects openly, private and sandbox ones only with the object's valid ?exp&sig
++// (from /download), which a signed playlist carries onto each URI; deleted 410, not ready 404. Off, these paths 404.
++const HLS_TYPE = 'application/vnd.apple.mpegurl';
++const SEGMENT_NAME = /^(init\.mp4|\d{6,}\.m4s)$/;
++
++/** The object of an HLS request, or null when answered (or passed on to the 404 while the flag is off). */
++async function hlsObject(req, res, next) {
++ if (!config.hls.enabled) { next(); return null; }
++ if (require('../drill').refuseBytes(res)) return null;
++ const obj = await model.getObject(String(req.params.id || ''));
++ if (!obj) { res.status(404).json({ error: 'Not found' }); return null; }
++ const signed = !!req.query.sig && signing.verifyDownload(obj.id, req.query.exp, req.query.sig);
++ const closed = obj.visibility === 'private' || await db.isSandboxTenant(obj.app_id);
++ if (closed && !signed) { res.status(404).json({ error: 'Not found' }); return null; }
++ if (obj.lifecycle_status === 'deleted') { res.status(410).json({ error: 'Gone' }); return null; }
++ if (obj.lifecycle_status !== 'ready') { res.status(404).json({ error: 'Not found' }); return null; }
++ res.set({
++ 'X-Content-Type-Options': 'nosniff', 'X-Robots-Tag': 'noindex', 'Access-Control-Allow-Origin': '*',
++ 'Cache-Control': closed ? 'private, no-store' : 'public, max-age=300',
++ });
++ return { obj, closed, query: signed ? `exp=${encodeURIComponent(req.query.exp)}&sig=${encodeURIComponent(req.query.sig)}` : '' };
++}
++
++const hlsRoute = (fn) => async (req, res, next) => {
++ try {
++ const h = await hlsObject(req, res, next);
++ if (h) await fn(req, res, h);
++ } catch (err) {
++ console.error('[Objects] HLS error:', err.message);
++ if (!res.headersSent) res.status(500).json({ error: 'Failed to serve the playlist' });
++ }
++};
++
++publicRouter.get('/:id/master.m3u8', hlsRoute(async (req, res, { obj, query }) => {
++ const timeline = require('./timeline');
++ const rows = await timeline.list(obj.id, timeline.SOURCE);
++ if (!rows.some((r) => Number(r.seq) > 0)) return res.status(404).json({ error: 'No timeline' });
++ res.type(HLS_TYPE).send(timeline.masterPlaylist([{ name: timeline.SOURCE, rows }], { query }));
++}));
++
++publicRouter.get('/:id/source/index.m3u8', hlsRoute(async (req, res, { obj, query }) => {
++ const timeline = require('./timeline');
++ const rows = await timeline.list(obj.id, timeline.SOURCE);
++ if (!rows.some((r) => Number(r.seq) > 0)) return res.status(404).json({ error: 'No timeline' });
++ res.type(HLS_TYPE).send(timeline.mediaPlaylist(rows, { query }));
++}));
++
++publicRouter.get('/:id/source/:name', hlsRoute(async (req, res, { obj, closed }) => {
++ const timeline = require('./timeline');
++ const name = String(req.params.name || '');
++ const row = SEGMENT_NAME.test(name) ? await timeline.byName(obj.id, timeline.SOURCE, name) : null;
++ if (!row) return res.status(404).json({ error: 'Not found' });
++ const mime = Number(row.seq) === 0 ? 'video/mp4' : 'video/iso.segment';
++ // This node's copy first; otherwise a short presigned URL to the durable one.
++ if (row.local_path && fs.existsSync(row.local_path)) {
++ return require('../public/routes').streamFileWithRange(req, res, row.local_path, {
++ 'Content-Type': mime, 'Cache-Control': closed ? 'private, no-store' : 'public, max-age=3600',
++ });
++ }
++ const url = row.durable_provider
++ ? await require('../vod/vod-storage').presignGet(row.durable_provider, row.key, 300, { contentType: mime }).catch(() => null)
++ : null;
++ if (!url) return res.status(404).json({ error: 'Segment bytes unavailable' });
++ res.set('Cache-Control', 'private, max-age=0');
++ res.redirect(302, url);
++}));
publicRouter.get('/:id', async (req, res) => {
// A restore drill (MEDIA_DRILL) serves no stored bytes, local or by a B2/R2 redirect.
if (require('../drill').refuseBytes(res)) return;
ddiff --git a/server/objects/timeline.js b/server/objects/timeline.js
nnew file mode 100644
iindex 0000000..c91f957
---- /dev/null
++++ b/server/objects/timeline.js
@@@ -0,0 +1,148 @@
++/**
++ * OpenVibe.Media — the timeline index (media_timeline; docs/media-fabric.md §3, F3.1).
++ *
++ * A video object is a timeline of CMAF segments per rendition. This module is the store: it answers
++ * "the bytes for 01:47:12.350" (segmentAt) and "the segments of a rendition in order" (segments) without
++ * opening a file, and writes the playlists from the rows. Segment 0 of a rendition is its init segment.
++ *
++ * Bytes: <object>/<rendition>/<name> is the segment's key; a copy on this node's disk sits at
++ * OBJECTS_PATH/.timeline/<app>/<object>/<rendition>/<name> (local_path), and a durable one under the same key on
++ * durable_provider. The object.cmaf job (server/jobs/cmaf.js) writes the source rendition; the public routes
++ * /o/:id/master.m3u8 and /o/:id/source/… (objects/routes.js) serve it behind MEDIA_HLS_ENABLED.
++ */
++'use strict';
++
++const fs = require('fs');
++const path = require('path');
++const config = require('../config');
++const db = require('../db/database');
++
++const SOURCE = 'source';
++const INIT_NAME = 'init.mp4';
++const FIELDS = ['name', 'start_ms', 'end_ms', 'keyframe_ms', 'key', 'local_path', 'durable_provider', 'packed_object_id', 'byte_offset', 'byte_length', 'sha256', 'durability'];
++
++function localRoot() { return path.join(path.resolve(config.objects.path), '.timeline'); }
++function localPathFor(obj, rendition, name) { return path.join(localRoot(), obj.app_id, obj.id, rendition, name); }
++function keyFor(objectId, rendition, name) { return `${objectId}/${rendition}/${name}`; }
++function segmentName(seq) { return seq === 0 ? INIT_NAME : `${String(seq).padStart(6, '0')}.m4s`; }
++
++/** Every row of a rendition in order, the init segment (seq 0) first. */
++async function list(objectId, rendition = SOURCE) {
++ return await db.all('SELECT * FROM media_timeline WHERE object_id = ? AND rendition = ? ORDER BY seq', [objectId, rendition]);
++}
++
++/** The media segments of a rendition in order (no init segment). */
++async function segments(objectId, rendition = SOURCE) {
++ return await db.all('SELECT * FROM media_timeline WHERE object_id = ? AND rendition = ? AND seq > 0 ORDER BY seq', [objectId, rendition]);
++}
++
++async function byName(objectId, rendition, name) {
++ return await db.get('SELECT * FROM media_timeline WHERE object_id = ? AND rendition = ? AND name = ?', [objectId, rendition, name]) || null;
++}
++
++async function has(objectId, rendition = SOURCE) {
++ return !!(await db.get('SELECT 1 AS x FROM media_timeline WHERE object_id = ? AND rendition = ? AND seq > 0 LIMIT 1', [objectId, rendition]));
++}
++
++/** The segment holding the instant tMs (milliseconds from the start): start_ms <= t < end_ms, or null past either end. */
++async function segmentAt(objectId, rendition, tMs) {
++ const t = Math.floor(Number(tMs));
++ if (!Number.isFinite(t) || t < 0) return null;
++ const row = await db.get(`SELECT * FROM media_timeline WHERE object_id = ? AND rendition = ? AND seq > 0 AND start_ms <= ?
++ ORDER BY start_ms DESC, seq DESC LIMIT 1`, [objectId, rendition, t]);
++ return row && t < Number(row.end_ms) ? row : null;
++}
++
++function same(a, b) {
++ return FIELDS.every((f) => (a[f] == null ? null : String(a[f])) === (b[f] == null ? null : String(b[f])));
++}
++
++/**
++ * Make the rows of one rendition exactly `rows` (each { seq, name, start_ms, … }), in one transaction: a row that already
++ * says the same is not written, a changed one is updated, rows past the new end are dropped. → { inserted, updated, unchanged, removed }.
++ */
++async function replace(objectId, rendition, rows, { jobId = null } = {}) {
++ const out = { inserted: 0, updated: 0, unchanged: 0, removed: 0 };
++ await db.getDb().tx(async () => {
++ const before = new Map((await list(objectId, rendition)).map((r) => [Number(r.seq), r]));
++ for (const r of rows) {
++ const prev = before.get(r.seq);
++ if (prev && same(prev, r)) { out.unchanged++; continue; }
++ await db.run(`INSERT INTO media_timeline (object_id, rendition, seq, ${FIELDS.join(', ')}, job_id)
++ VALUES (?, ?, ?, ${FIELDS.map(() => '?').join(', ')}, ?)
++ ON CONFLICT (object_id, rendition, seq) DO UPDATE SET ${FIELDS.map((f) => `${f} = excluded.${f}`).join(', ')},
++ job_id = excluded.job_id, updated_at = ov_now()`,
++ [objectId, rendition, r.seq, ...FIELDS.map((f) => r[f] ?? null), jobId]);
++ out[prev ? 'updated' : 'inserted']++;
++ }
++ const last = rows.reduce((m, r) => Math.max(m, r.seq), -1);
++ out.removed = (await db.run('DELETE FROM media_timeline WHERE object_id = ? AND rendition = ? AND seq > ?', [objectId, rendition, last])).changes || 0;
++ });
++ return out;
++}
++
++/**
++ * The object's whole timeline is gone with its bytes (a purge, or a vod/clip deleted for good): local files, durable
++ * copies (best effort), then the rows. Callers have already refused a held object. → rows removed.
++ */
++async function removeObject(objectId) {
++ if (!objectId) return 0;
++ const rows = await db.all('SELECT * FROM media_timeline WHERE object_id = ?', [objectId]);
++ if (!rows.length) return 0;
++ const root = localRoot() + path.sep;
++ const vodStorage = require('../vod/vod-storage');
++ const dirs = new Set();
++ for (const r of rows) {
++ if (r.local_path && path.resolve(r.local_path).startsWith(root)) {
++ try { fs.unlinkSync(r.local_path); } catch { /* already gone */ }
++ dirs.add(path.dirname(r.local_path));
++ }
++ if (r.durable_provider && vodStorage.providerConfigured(r.durable_provider)) {
++ await vodStorage.deleteObject(r.durable_provider, r.key).catch((err) => console.warn(`[Timeline] ${r.durable_provider}:${r.key} not deleted: ${err.message}`));
++ }
++ }
++ for (const d of dirs) {
++ for (const p of [d, path.dirname(d)]) { try { fs.rmdirSync(p); } catch { /* not empty or gone */ } }
++ }
++ return (await db.run('DELETE FROM media_timeline WHERE object_id = ?', [objectId])).changes || 0;
++}
++
++// ── Playlists (from the rows, never from files) ──────────────
++
++const secs = (ms) => (Number(ms) / 1000).toFixed(3);
++
++/** `?exp=…&sig=…` carried onto every URI of a signed playlist, so a private object's segments pass the same check. */
++function withQuery(uri, query) { return query ? `${uri}?${query}` : uri; }
++
++/** The media playlist of a finished rendition: VOD, ENDLIST, one EXTINF per segment with its timeline duration. */
++function mediaPlaylist(rows, { query = '' } = {}) {
++ const segs = rows.filter((r) => Number(r.seq) > 0);
++ const init = rows.find((r) => Number(r.seq) === 0);
++ const target = Math.max(1, ...segs.map((r) => Math.ceil((Number(r.end_ms) - Number(r.start_ms)) / 1000)));
++ const lines = ['#EXTM3U', '#EXT-X-VERSION:7', `#EXT-X-TARGETDURATION:${target}`, `#EXT-X-MEDIA-SEQUENCE:${segs.length ? Number(segs[0].seq) : 1}`,
++ '#EXT-X-PLAYLIST-TYPE:VOD', '#EXT-X-INDEPENDENT-SEGMENTS'];
++ if (init) lines.push(`#EXT-X-MAP:URI="${withQuery(init.name, query)}"`);
++ for (const r of segs) lines.push(`#EXTINF:${secs(Number(r.end_ms) - Number(r.start_ms))},`, withQuery(r.name, query));
++ lines.push('#EXT-X-ENDLIST', '');
++ return lines.join('\n');
++}
++
++/** The master playlist: one variant per rendition (only `source` in F3.1), BANDWIDTH = its peak segment bitrate. */
++function masterPlaylist(renditions, { query = '' } = {}) {
++ const lines = ['#EXTM3U', '#EXT-X-VERSION:7', '#EXT-X-INDEPENDENT-SEGMENTS'];
++ for (const { name, rows } of renditions) {
++ const segs = rows.filter((r) => Number(r.seq) > 0);
++ const rate = (r) => Math.ceil(Number(r.byte_length) * 8 * 1000 / Math.max(1, Number(r.end_ms) - Number(r.start_ms)));
++ const peak = Math.max(1, ...segs.map(rate));
++ const total = segs.reduce((a, r) => a + Number(r.end_ms) - Number(r.start_ms), 0);
++ const avg = Math.max(1, Math.ceil(segs.reduce((a, r) => a + Number(r.byte_length), 0) * 8 * 1000 / Math.max(1, total)));
++ lines.push(`#EXT-X-STREAM-INF:BANDWIDTH=${peak},AVERAGE-BANDWIDTH=${avg}`, withQuery(`${name}/index.m3u8`, query));
++ }
++ lines.push('');
++ return lines.join('\n');
++}
++
++module.exports = {
++ SOURCE, INIT_NAME, localRoot, localPathFor, keyFor, segmentName,
++ list, segments, byName, has, segmentAt, replace, removeObject, mediaPlaylist, masterPlaylist,
++};
ddiff --git a/server/vod/orphans-report.js b/server/vod/orphans-report.js
iindex b45fd1a..478893f 100644
---- a/server/vod/orphans-report.js
++++ b/server/vod/orphans-report.js
@@@ -222,6 +222,8 @@ async function localReferences() {
for (const l of await db.all(`${LOCATION_ROWS} WHERE l.provider = 'local'`)) {
if (locationWanted(l)) add(l.key); else deleted.set(path.resolve(String(l.key)), l);
}
++ // CMAF segments on this node's disk (OBJECTS_PATH/.timeline/…), each named by its media_timeline row (F3.1).
++ for (const t of await db.all('SELECT local_path FROM media_timeline WHERE local_path IS NOT NULL')) add(t.local_path);
const thumb = (u) => { const name = model.thumbFileFromUrl(u); if (name) add(path.join(config.thumbnails.path, name)); };
const segmentBases = new Set(); // browser-chunk segments <base>.seg-<n>-<ms>.webm belong to <base>.webm
for (const v of await db.all('SELECT id, file_path, master_file_path, thumbnail_url FROM vods')) {
@@@ -305,6 +307,7 @@ async function remoteReferences(provider) {
if (l.bucket && bucket && l.bucket !== bucket) continue; // a copy recorded in another bucket
if (locationWanted(l)) wanted.add(l.key); else deleted.set(l.key, l);
}
++ for (const t of await db.all('SELECT key FROM media_timeline WHERE durable_provider = ?', [provider])) wanted.add(t.key); // durable CMAF segments
const rows = new Map(); // every key a vods/clips row points at, with the row (stale R2 copies are recognised by it)
for (const table of ['vods', 'clips']) {
for (const r of await db.all(`SELECT id, file_path, storage_provider, storage_key FROM ${table} WHERE COALESCE(file_path, '') != ''`)) {
ddiff --git a/server/vod/vod-storage.js b/server/vod/vod-storage.js
iindex dfbd8c6..f39d6db 100644
---- a/server/vod/vod-storage.js
++++ b/server/vod/vod-storage.js
@@@ -825,6 +825,8 @@ async function deleteVodObjects(vod) {
if (!vod?.file_path) return;
// Last line of defence for every delete path: a held object keeps its bytes.
if (await _held(vod)) { console.warn(`[VodStorage] Not deleting media of ${vod.object_id}: under a retention hold`); return; }
++ // Its CMAF segments and timeline rows (F3.1, objects/timeline.js) go with it.
++ if (vod.object_id) await require('../objects/timeline').removeObject(vod.object_id).catch((err) => console.warn(`[VodStorage] timeline of ${vod.object_id}: ${err.message}`));
const local = localPathForVod(vod);
if (fs.existsSync(local)) {
try { fs.unlinkSync(local); } catch { /* ignore */ }
ddiff --git a/test/media-timeline.test.js b/test/media-timeline.test.js
nnew file mode 100644
iindex 0000000..a137a69
---- /dev/null
++++ b/test/media-timeline.test.js
@@@ -0,0 +1,239 @@
++'use strict';
++// Segment-native video, first slice (F3.1; docs/media-fabric.md §3): object.cmaf stream-copies a finished video into
++// CMAF segments on its own keyframes and writes one media_timeline row per segment; the HLS playlists come from the
++// rows (ENDLIST, the rows' durations); segmentAt answers the segment at an instant, at the boundaries too; a rerun is a
++// no-op; a private object's playlist refuses an anonymous reader and its signed one carries the signature on; with
++// MEDIA_HLS_ENABLED off nothing answers; a purge removes the segments unless the object is held.
++const assert = require('assert');
++const fs = require('fs');
++const os = require('os');
++const path = require('path');
++const http = require('http');
++const crypto = require('crypto');
++const { spawn, spawnSync } = require('child_process');
++(async () => {
++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-timeline-'));
++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
++ process.env.VOD_PATH = dir('vods');
++ process.env.CLIPS_PATH = dir('clips');
++ process.env.FILES_PATH = dir('files');
++ process.env.THUMBNAILS_PATH = dir('thumbnails');
++ process.env.PASTES_PATH = dir('pastes');
++ process.env.OBJECTS_PATH = dir('objects');
++ process.env.MEDIA_JOBS_POLL_MS = '50';
++ process.env.MEDIA_INVARIANT_SCAN_HOURS = '0';
++ process.env.MEDIA_UPLOAD_MIN_FREE_MB = '0';
++ process.env.MEDIA_HLS_ENABLED = '1';
++ for (const p of ['B2', 'R2']) for (const k of ['ENDPOINT', 'BUCKET', 'KEY_ID', 'APP_KEY', 'ACCESS_KEY_ID', 'SECRET_ACCESS_KEY']) process.env[`MEDIA_${p}_${k}`] = '';
++ const stub = http.createServer((req, res) => {
++ req.resume();
++ req.on('end', () => {
++ res.setHeader('Content-Type', 'application/json');
++ if (req.url === '/oauth/token') return res.end(JSON.stringify({ access_token: 'tok', token_type: 'Bearer', expires_in: 300 }));
++ res.statusCode = 404; res.end('{}');
++ });
++ });
++ const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
++ const waitFor = async (fn, ms = 60000) => { const end = Date.now() + ms; while (Date.now() < end) { if (await fn()) return true; await sleep(25); } return false; };
++ const sha = (buf) => crypto.createHash('sha256').update(buf).digest('hex');
++ const hasFfmpeg = spawnSync('ffmpeg', ['-version']).status === 0;
++
++ (async () => {
++ const config = require('../server/config');
++ const timeline = require('../server/objects/timeline');
++ const { parsePlaylist } = require('../server/jobs/cmaf');
++ assert.strictEqual(config.hls.enabled, true);
++
++ // ── The store alone: segmentAt and the playlists over hand-made rows ──
++ assert.deepStrictEqual(parsePlaylist('#EXTM3U\n#EXT-X-MAP:URI="init.mp4"\n#EXTINF:3.000000,\n000001.m4s\n#EXTINF:1.5,\n000002.m4s\n#EXT-X-ENDLIST\n'),
++ [{ name: '000001.m4s', seconds: 3 }, { name: '000002.m4s', seconds: 1.5 }]);
++ const row = (seq, start, end, len = 1000) => ({ seq, name: timeline.segmentName(seq), start_ms: start, end_ms: end, keyframe_ms: seq ? start : null,
++ key: `med_HAND/source/${timeline.segmentName(seq)}`, local_path: null, durable_provider: null, packed_object_id: null, byte_offset: null,
++ byte_length: len, sha256: 'a'.repeat(64), durability: 'local' });
++ const hand = [row(0, 0, 0, 500), row(1, 0, 3000, 30000), row(2, 3000, 4500), row(3, 4500, 6432)];
++ assert.deepStrictEqual(await timeline.replace('med_HAND', 'source', hand), { inserted: 4, updated: 0, unchanged: 0, removed: 0 });
++ const at = async (t) => { const r = await timeline.segmentAt('med_HAND', 'source', t); return r ? Number(r.seq) : null; };
++ assert.deepStrictEqual([await at(0), await at(2999), await at(3000), await at(4499), await at(4500), await at(6431), await at(6432), await at(-1), await at(NaN)],
++ [1, 1, 2, 2, 3, 3, null, null, null], 'start inclusive, end exclusive; nothing past either end');
++ assert.strictEqual(await timeline.segmentAt('med_HAND', '720p', 10), null, 'another rendition has no rows');
++ const mp = timeline.mediaPlaylist(await timeline.list('med_HAND'));
++ assert.ok(mp.includes('#EXT-X-TARGETDURATION:3\n') && mp.includes('#EXT-X-MAP:URI="init.mp4"') && mp.includes('#EXTINF:1.932,\n000003.m4s') && mp.trim().endsWith('#EXT-X-ENDLIST'), mp);
++ assert.ok(timeline.masterPlaylist([{ name: 'source', rows: hand }]).includes('#EXT-X-STREAM-INF:BANDWIDTH=80000,AVERAGE-BANDWIDTH='), 'peak = 30000 B over 3 s');
++ assert.deepStrictEqual(await timeline.replace('med_HAND', 'source', hand.slice(0, 2)), { inserted: 0, updated: 0, unchanged: 2, removed: 2 }, 'rows past a shorter end are dropped');
++ assert.strictEqual(await timeline.removeObject('med_HAND'), 2);
++ if (!hasFfmpeg) { console.log('media timeline: skipped (ffmpeg not found)'); process.exit(0); }
++
++ await new Promise((r) => stub.listen(0, '127.0.0.1', r));
++ const base = `http://127.0.0.1:${stub.address().port}`;
++ const db = require('../server/db/database');
++ await db.upsertApp({ app_id: 'live', api_key: 'live-key' });
++ const events = require('../server/events');
++ events.init({ eventsUrl: base, clientSecret: 's', networkUrl: base, intervalMs: 60000 });
++ const model = require('../server/objects/model');
++ const queue = require('../server/jobs/queue');
++ const worker = require('../server/jobs/worker');
++ const express = require('express');
++ const app = express();
++ app.use(express.json());
++ app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));
++ app.use('/api/v2/:app/objects', require('../server/objects/routes'));
++ app.use('/o', require('../server/objects/routes').publicRouter);
++ const server = http.createServer(app);
++ await new Promise((r) => server.listen(0, '127.0.0.1', r));
++ const api = `http://127.0.0.1:${server.address().port}`;
++ const call = async (method, p, { body } = {}) => {
++ const res = await fetch(api + p, { method, headers: { authorization: 'Bearer live-key', 'content-type': 'application/json' }, body: body === undefined ? undefined : JSON.stringify(body) });
++ return { status: res.status, body: await res.json().catch(() => null) };
++ };
++ const get = async (url, headers = {}) => {
++ const res = await fetch(url.startsWith('http') ? url : api + url, { headers, redirect: 'manual' });
++ return { status: res.status, headers: res.headers, buf: Buffer.from(await res.arrayBuffer()) };
++ };
++ await worker.start();
++
++ // A 7 s source, 10 fps, a keyframe every 1.5 s: segments cut at 3.0, 4.5, 6.0 for a 2 s target.
++ const src = path.join(process.env.VOD_PATH, 'src.mp4');
++ const mk = spawnSync('ffmpeg', ['-v', 'error', '-y', '-f', 'lavfi', '-i', 'testsrc=duration=7:size=160x90:rate=10', '-f', 'lavfi', '-i', 'sine=frequency=440:duration=7',
++ '-c:v', 'mpeg4', '-g', '15', '-c:a', 'aac', '-shortest', src]);
++ assert.strictEqual(mk.status, 0, String(mk.stderr));
++ const native = async (visibility) => {
++ const file = path.join(process.env.VOD_PATH, `${visibility}.mp4`);
++ fs.copyFileSync(src, file);
++ const id = await model.createObject({ app_id: 'live', owner_user_id: 5, visibility, lifecycle_status: 'ready', kind: 'file', mime_type: 'video/mp4', size_bytes: fs.statSync(file).size });
++ await model.upsertLocation(id, { provider: 'local', key: file, state: 'present', size_bytes: fs.statSync(file).size, verified: true });
++ return id;
++ };
++ const pub = await native('public');
++ const priv = await native('private');
++ const before = JSON.stringify([await model.getObject(pub), await model.listLocations(pub)]);
++
++ const runJob = async (objectId, params) => {
++ const r = await call('POST', '/api/v2/live/jobs', { body: { type: 'object.cmaf', object_id: objectId, ...(params && { params }) } });
++ assert.strictEqual(r.status, 202, JSON.stringify(r.body));
++ const fin = await waitFor(async () => ['succeeded', 'failed'].includes((await queue.get(r.body.job.id)).status));
++ assert.ok(fin, `object.cmaf finishes: ${JSON.stringify(queue.jobPublic(await queue.get(r.body.job.id)))}`);
++ return queue.jobPublic(await queue.get(r.body.job.id));
++ };
++
++ // ── The job → contiguous rows with the right sha256 ──
++ const j = await runJob(pub, { segment_seconds: 2 });
++ assert.strictEqual(j.status, 'succeeded', JSON.stringify(j.error));
++ assert.strictEqual(j.result.segments, 4, JSON.stringify(j.result));
++ assert.deepStrictEqual([j.result.durable, j.result.local_only, j.result.uploaded, j.result.rows.inserted], [0, 5, 0, 5], 'no B2 here: every row is local');
++ const rows = await timeline.list(pub);
++ assert.deepStrictEqual(rows.map((r) => Number(r.seq)), [0, 1, 2, 3, 4]);
++ assert.deepStrictEqual(rows.map((r) => r.name), ['init.mp4', '000001.m4s', '000002.m4s', '000003.m4s', '000004.m4s']);
++ const segs = rows.slice(1);
++ assert.strictEqual(Number(segs[0].start_ms), 0);
++ for (let i = 1; i < segs.length; i++) assert.strictEqual(Number(segs[i].start_ms), Number(segs[i - 1].end_ms), `segment ${i + 1} starts where ${i} ends`);
++ assert.deepStrictEqual(segs.map((r) => [Number(r.start_ms), Number(r.end_ms), Number(r.keyframe_ms)]), [[0, 3000, 0], [3000, 4500, 3000], [4500, 6000, 4500], [6000, 7000, 6000]], 'cut on the source keyframes');
++ for (const r of rows) {
++ const bytes = fs.readFileSync(r.local_path);
++ assert.strictEqual(sha(bytes), r.sha256, `${r.name} sha256`);
++ assert.strictEqual(bytes.length, Number(r.byte_length));
++ assert.strictEqual(r.key, `${pub}/source/${r.name}`);
++ assert.strictEqual(r.durability, 'local');
++ assert.ok(r.local_path.startsWith(path.join(path.resolve(process.env.OBJECTS_PATH), '.timeline', 'live', pub, 'source')));
++ }
++ assert.strictEqual(JSON.stringify([await model.getObject(pub), await model.listLocations(pub)]), before, 'the source is untouched');
++ assert.strictEqual(fs.readdirSync(path.join(process.env.OBJECTS_PATH, '.jobs')).length, 0, 'the work directory is removed');
++
++ // ── The playlists come from the rows ──
++ const master = await get(`/o/${pub}/master.m3u8`);
++ assert.strictEqual(master.status, 200);
++ assert.ok(/^application\/vnd\.apple\.mpegurl/.test(master.headers.get('content-type')));
++ assert.ok(/#EXT-X-STREAM-INF:BANDWIDTH=\d+,AVERAGE-BANDWIDTH=\d+\nsource\/index\.m3u8\n/.test(master.buf.toString()), master.buf.toString());
++ const media = (await get(`/o/${pub}/source/index.m3u8`)).buf.toString();
++ assert.ok(media.trim().endsWith('#EXT-X-ENDLIST') && media.includes('#EXT-X-PLAYLIST-TYPE:VOD') && media.includes('#EXT-X-TARGETDURATION:3\n'), media);
++ assert.deepStrictEqual(parsePlaylist(media), [['000001.m4s', 3], ['000002.m4s', 1.5], ['000003.m4s', 1.5], ['000004.m4s', 1]].map(([name, seconds]) => ({ name, seconds })), 'the durations are the rows');
++ for (const r of rows) {
++ const s = await get(`/o/${pub}/source/${r.name}`);
++ assert.strictEqual(s.status, 200, r.name);
++ assert.strictEqual(sha(s.buf), r.sha256, `${r.name} served as indexed`);
++ }
++ assert.strictEqual((await get(`/o/${pub}/source/000002.m4s`, { range: 'bytes=0-99' })).status, 206, 'segments take ranges');
++ assert.strictEqual((await get(`/o/${pub}/source/000009.m4s`)).status, 404);
++ assert.strictEqual((await get(`/o/${pub}/source/..%2Findex.m3u8`)).status, 404);
++ // A real HLS reader agrees: the playlist plays back as the 7 s source.
++ // (spawned, not spawnSync: the server answering it runs in this process)
++ const probe = await new Promise((resolve) => {
++ const p = spawn('ffprobe', ['-v', 'error', '-rw_timeout', '10000000', '-show_entries', 'format=duration', '-of', 'csv=p=0', `${api}/o/${pub}/master.m3u8`]);
++ let out = '';
++ p.stdout.on('data', (d) => { out += d; });
++ p.stderr.on('data', (d) => { out += d; });
++ p.on('close', () => resolve(out.trim()));
++ });
++ assert.ok(Math.abs(Number(probe) - 7) < 0.2, `ffprobe reads the playlist: ${probe}`);
++
++ // ── segmentAt on the real timeline, at the boundaries ──
++ const seqAt = async (t) => { const r = await timeline.segmentAt(pub, 'source', t); return r ? Number(r.seq) : null; };
++ assert.deepStrictEqual([await seqAt(0), await seqAt(2999), await seqAt(3000), await seqAt(4500), await seqAt(5999), await seqAt(6000), await seqAt(6999), await seqAt(7000)],
++ [1, 1, 2, 3, 3, 4, 4, null]);
++
++ // ── A rerun is a no-op: same rows, no file rewritten ──
++ const mtimes = rows.map((r) => fs.statSync(r.local_path).mtimeMs);
++ const j2 = await runJob(pub, { segment_seconds: 2 });
++ assert.strictEqual(j2.status, 'succeeded', JSON.stringify(j2.error));
++ assert.deepStrictEqual(j2.result.rows, { inserted: 0, updated: 0, unchanged: 5, removed: 0 });
++ assert.strictEqual(j2.result.uploaded, 0);
++ assert.strictEqual(JSON.stringify(await timeline.list(pub)), JSON.stringify(rows), 'the rows are as they were (job_id, updated_at too)');
++ assert.deepStrictEqual(rows.map((r) => fs.statSync(r.local_path).mtimeMs), mtimes, 'no segment rewritten');
++
++ // ── The job's abort signal: nothing written, the work directory removed ──
++ const cmaf = require('../server/jobs/cmaf');
++ const ac = new AbortController();
++ ac.abort(new Error('cancelled in test'));
++ await assert.rejects(cmaf.spec.run({ id: 'mjob_ABORTED', app_id: 'live', object_id: priv, params: {} }, { signal: ac.signal }), /cancelled in test/);
++ assert.strictEqual(await timeline.has(priv), false);
++ assert.strictEqual(fs.existsSync(path.join(process.env.OBJECTS_PATH, '.jobs', 'mjob_ABORTED')), false);
++
++ // ── A private object: anonymous readers get nothing; the signed playlist carries its signature on ──
++ assert.strictEqual((await runJob(priv)).status, 'succeeded');
++ for (const p of ['master.m3u8', 'source/index.m3u8', 'source/init.mp4', 'source/000001.m4s']) assert.strictEqual((await get(`/o/${priv}/${p}`)).status, 404, `anonymous ${p}`);
++ const dl = await call('GET', `/api/v2/live/objects/${priv}/download?format=json`);
++ assert.strictEqual(dl.status, 200);
++ const signed = new URL(dl.body.hls_url);
++ assert.strictEqual(signed.pathname, `/o/${priv}/master.m3u8`);
++ const sm = await get(signed.pathname + signed.search);
++ assert.strictEqual(sm.status, 200);
++ assert.strictEqual(sm.headers.get('cache-control'), 'private, no-store');
++ assert.ok(sm.buf.toString().includes(`source/index.m3u8${signed.search}`), 'the variant URI is signed');
++ const smedia = (await get(`/o/${priv}/source/index.m3u8${signed.search}`)).buf.toString();
++ assert.ok(smedia.includes(`#EXT-X-MAP:URI="init.mp4${signed.search}"`) && smedia.includes(`000001.m4s${signed.search}`), smedia);
++ assert.strictEqual((await get(`/o/${priv}/source/000001.m4s${signed.search}`)).status, 200);
++ assert.strictEqual((await get(`/o/${priv}/master.m3u8?exp=${signed.searchParams.get('exp')}&sig=${'0'.repeat(64)}`)).status, 404, 'a forged signature');
++ assert.strictEqual((await get(`/o/${pub}/master.m3u8${signed.search}`)).status, 200, 'a public object needs no signature');
++ assert.strictEqual((await call('GET', `/api/v2/live/objects/${pub}/download?format=json`)).body.hls_url, `${config.publicUrl}/o/${pub}/master.m3u8`);
++
++ // ── MEDIA_HLS_ENABLED off: the routes 404, the job is refused, the download answer is as before ──
++ config.hls.enabled = false;
++ assert.strictEqual((await get(`/o/${pub}/master.m3u8`)).status, 404);
++ assert.strictEqual((await get(`/o/${pub}/source/000001.m4s`)).status, 404);
++ const off = await call('POST', '/api/v2/live/jobs', { body: { type: 'object.cmaf', object_id: pub } });
++ assert.deepStrictEqual([off.status, off.body.code], [409, 'media.hls.disabled']);
++ assert.deepStrictEqual(Object.keys((await call('GET', `/api/v2/live/objects/${pub}/download?format=json`)).body).sort(), ['expires_at', 'public', 'url']);
++ config.hls.enabled = true;
++
++ // ── Deleted: 410; a held object keeps its segments through a purge, an unheld one loses them ──
++ const pubPaths = rows.map((r) => r.local_path);
++ await model.softDelete(await model.getObject(pub));
++ assert.strictEqual((await get(`/o/${pub}/master.m3u8`)).status, 410);
++ await model.softDelete(await model.getObject(priv));
++ const hold = await model.placeHold({ object_id: priv, kind: 'admin', reason: 'test' });
++ await model.purgeExpired({ retentionDays: 0 });
++ assert.strictEqual(await timeline.has(pub), false, 'a purge removes the timeline');
++ assert.ok(pubPaths.every((p) => !fs.existsSync(p)), 'and its segment files');
++ assert.strictEqual(await timeline.has(priv), true, 'a held object keeps its timeline');
++ assert.ok((await timeline.list(priv)).every((r) => fs.existsSync(r.local_path)));
++ await model.releaseHold(hold.id);
++ await model.purgeExpired({ retentionDays: 0 });
++ assert.strictEqual(await timeline.has(priv), false);
++
++ worker.stop();
++ events._reset();
++ server.close();
++ stub.close();
++ console.log('media-timeline: all checks passed');
++ process.exit(0);
++ })().catch((err) => { console.error(err); process.exit(1); });
++})().catch((err) => { console.error(err); process.exit(1); });
ddiff --git a/test/storage-orphans.test.js b/test/storage-orphans.test.js
iindex a5de097..dff2d03 100644
---- a/test/storage-orphans.test.js
++++ b/test/storage-orphans.test.js
@@@ -113,6 +113,11 @@ const crypto = require('crypto');
VALUES ('mup_OPEN', ?, 'live', 8, 16, 2, 'active', datetime('now', '-1 hour'))`).run(kept);
await raw.prepare("INSERT INTO media_upload_parts (upload_id, part_number, size_bytes, sha256) VALUES ('mup_OPEN', 1, 8, 'x')").run();
write(path.join(env.OBJECTS_PATH, '.parts', 'mup_OPEN'), '1', 8, 3 * HOUR);
++ // A CMAF segment named by its media_timeline row (F3.1), on this disk and on B2: wanted in both places.
++ const seg = write(path.join(env.OBJECTS_PATH, '.timeline', 'live', kept, 'source'), '000001.m4s', 40, 3 * HOUR);
++ buckets['b2-bucket'].set(`${kept}/source/000001.m4s`, Buffer.alloc(40));
++ await require('../server/objects/timeline').replace(kept, 'source', [{ seq: 1, name: '000001.m4s', start_ms: 0, end_ms: 2000, keyframe_ms: 0,
++ key: `${kept}/source/000001.m4s`, local_path: seg, durable_provider: 'b2', byte_length: 40, sha256: 'b'.repeat(64), durability: 'durable' }]);
// ── What storage holds that no row names ──
write(env.VOD_PATH, 'vod-live-99-1758000000099.webm', 1000); // the recorder's name; row 99 never existed
@@@ -160,7 +165,7 @@ const crypto = require('crypto');
assert.ok(/vods-orphans-report/.test(remote['b2:vods-orphans/vod-live-7-1758000000007.webm'].hint), 'parked recordings point at the prefix report');
assert.strictEqual(remote['r2:vods/vod-live-4-1758000000004.webm'].hint, 'an R2 copy of vod 4, which is served from b2');
assert.deepStrictEqual(report.totals.unreferenced_remote, { b2: { keys: 2, bytes: 150 }, r2: { keys: 1, bytes: 50 } });
-- assert.deepStrictEqual([report.scope.providers.b2.listed, report.scope.providers.b2.keys, report.scope.providers.r2.keys], [true, 5, 2]);
++ assert.deepStrictEqual([report.scope.providers.b2.listed, report.scope.providers.b2.keys, report.scope.providers.r2.keys], [true, 6, 2], 'the CMAF segment is listed, not reported');
console.log('✅ unreferenced bucket keys: strays, parked recordings, an R2 copy left behind by a demotion');
// ── Copies the database records that are not there ──