openvibers/openvibe.media Commits

Materialized clips: a clip names its source's interior segments and re-encodes only its two edges, behind MEDIA_MATERIALIZED_CLIPS (off by default; F3.6) (#44)

OP OpenVibers committed 5 Oct 2026 · 706c2de95c · 18 files, +1272 −66

* Materialized clips: a clip names its source's interior segments and re-encodes only its two edges, behind MEDIA_MATERIALIZED_CLIPS (off by default; F3.6)

Implementation complete. Nothing committed or pushed; all changes are in the working tree of `agent/plan-b19-media-matclip-10051047`.

## Changes (file:line)

**Schema**
- `migrations/0005_materialized_clips.sql:1` — adds `init_name`, drops/re-adds the real `media_timeline_seq_check` (verified in `pg_constraint`) admitting exactly `init-head.mp4`/`init-tail.mp4` at negative seq; idempotent (DROP IF EXISTS + ADD).

**Readers**
- `server/objects/timeline.js:46` — `HEAD_INIT`/`TAIL_INIT`; `:48` `init_name` appended to `FIELDS`; `:127` `replace()` now also deletes negative-seq rows not in the new set; `:324` `mediaPlaylist()` resolves each segment's init (`init_name` else seq 0), emits `#EXT-X-MAP` for the first and `#EXT-X-DISCONTINUITY` + fresh MAP on every change (byte-for-byte identical when no edge inits); `:382` `clipRows()` refuses a virtual clip over a materialized one.
- `server/objects/routes.js:797` — `SEGMENT_NAME` accepts the two edge inits; `:901` every init row (`seq <= 0`) serves as `video/mp4`; `:700` `download?format=mp4` answers the HLS playlist for a timeline-only clip instead of queueing a remux.
- `server/jobs/previews.js:243` — `initRowFor()` (per-segment init by `init_name`, falling back to seq 0), used at `:265`/`:282` so sprites prepend each piece's own init.
- `server/objects/model.js:336` — clip projection stays `virtual: true` and preserves the `materialized: true` written by the job (readiness/`playableSql`/verify therefore unchanged).
- `server/vod/clip-materialize.js:55` `planWindow` (head/interior/tail, 100 ms boundary snap, 7-column location compare), `:213` `pinRows` (`SELECT … FOR SHARE` + `timeline.replace` in one transaction, retryable `media.timeline.changed` on a missing/moved source row), `:238` `materialize` (read source bytes via the sprite path, one-segment-per-edge fMP4 re-encode + its own init, content-addressed placement/staging/cleanup, stale-byte deletion).
- `server/vod/clip-jobs.js:86` materialize attempt inside `clip.cut` (flag + timeline + job context only), `:136` `markMaterialized` (ready row + `metadata.materialized` in the same transaction + `clip.ready`), `:281` passes `job`/`ctx` into the cut.
- `server/vod/clips-routes.js:458` clip DELETE calls `timeline.removeObject(clip.object_id)` for a `timeline` clip.

**Config/docs**
- `server/config.js:100` / `.env.example:155` — `MEDIA_MATERIALIZED_CLIPS` (default off).
- `docs/materialized-clips.md:3` (implemented), `docs/media-fabric.md:171` (new F3.6 line, still-open list, §4 materialized bullet, build order), `README.md:115`/`:278`/`:471` (jobs list, clip API, HLS routes), `STATUS.json:47`.
- `test/materialized-clip.test.js:1` — planWindow/mediaPlaylist/migration/`replace`/clipRows/`initRowFor`/pin races plus the ffmpeg e2e (packed source, serving, two discontinuities, ffprobe, readiness/verify, edge sprite, downloads, deletion both ways, flag-off/no-timeline).

## Test results
- Required set: `ov test test/materialized-clip.test.js test/clip-timeline.test.js test/media-timeline.test.js test/timeline-refs.test.js test/rendition-job.test.js test/pins.test.js` plus `test/clip-cut-job.test.js` → **7 passed in 52 s** (all ffmpeg/libx264 sections ran; the whole run and its ffmpeg children under `nice -n 15`).
- Grep-selected others, all passed: `clip-origin-filters`, `jobs-previews`, `object-first-writes`, `objects-model`, `outbox-transaction`, `placement-read-paths`, `private-not-found`, `readiness`, `retention-holds`, `security-idor`, `sec

* Materialized clips: a re-cut to a file drops the timeline and its edge bytes, no renditions for a clip without a file, the snapped end is stored, pending deletes are swept, and the pin locks in seq order (review)

**Findings**

1. **Materialized clip re-cut to a file** — the file-cut ready transition now deletes `metadata.materialized` in the same transaction (`server/vod/clip-jobs.js:125`) and calls `timeline.removeObject(clip.object_id)` after the commit (`server/vod/clip-jobs.js:133`); the DELETE route calls `removeObject` whenever `clip.object_id` is set, dropping the `storage_provider === 'timeline'` condition (`server/vod/clips-routes.js:459-460`). Guarded by "finding 1" checks (no rows, edge bytes gone, marker gone, stale `init-head.mp4` 404, `/c/:id` serves the file, plus DELETE of a non-timeline clip with planted rows).
2. **No rendition for a materialized clip** — `queueRendition` returns null when the clip object's metadata has `virtual === true` (`server/objects/timeline-queue.js:80,97`), and `hls.js` guards the lazy path the same way (`server/objects/hls.js:136`). Test: with renditions on, `queueRendition` and `discoveryForObject` queue no `rendition.create`.
3. **`end_time` drift** — `markMaterialized` stores `end_time = mat.end_ms / 1000` (`server/vod/clip-jobs.js:155`, call site `:91`) instead of `startTime + duration`. Test materializes a snap-back window and asserts `end_time` is the snapped end, not the drifted sum.
4. **Pending durable delete on clip delete** — the route logs `removeObject`'s `pending` (`server/vod/clips-routes.js:464`); retry is the codebase's existing pending contract from `purgeExpired` (`model.js:688-689`: keep the rows as the record and let a later sweep pass call `removeObject` again). I extended the **existing hourly object sweep** with `timeline.sweepPendingRemovals()` (`server/objects/timeline.js:250`, called at `server/index.js:288`), which finds deleted projected objects whose clips/vods row is gone and skips held ones. Test: forced durable-delete failure → warning logged, row survives, sweep drops it and its bytes.
5. **Pin deadlock** — added `ORDER BY seq` to the locking `FOR SHARE` select (`server/vod/clip-materialize.js:219`). Test asserts the ordering is present.

**Reviewer's note:** restored byte-for-byte the `EXT-X-MAP` for a rendition that has an init but no segments (`server/objects/timeline.js:357`), with an assertion in the playlist section.

**Tests:** all 7 mandated files pass — `materialized-clip`, `clip-timeline`, `media-timeline`, `rendition-job`, `timeline-queue`, `clip-cut-job`, `pins` (7 passed, 36 s, ffmpeg at nice 15). Changes are uncommitted and unpushed.

Changes

ddiff --git a/.env.example b/.env.example
iindex d89c769..f362542 100644
---- a/.env.example
++++ b/.env.example
@@@ -149,3 +149,7 @@ MEDIA_CAPABILITY_GATE=report
# Lifetime in seconds of a signed HLS playlist (download?format=json → hls_url of a private or sandbox object; its own
# signature purpose, accepted only by the HLS routes). Default 21600 (6 h), clamped to 60..43200.
#MEDIA_HLS_PLAYLIST_TTL_S=21600
++# Materialized clips (docs/materialized-clips.md, F3.6): 1 = with MEDIA_HLS_ENABLED and a source that has a source
++# timeline, clip.cut materializes the clip over the source's bytes — the interior rows name the source's segments and
++# only the two window edges are re-encoded; 0 (default) = every clip is cut with the full re-encode as before.
++#MEDIA_MATERIALIZED_CLIPS=0
ddiff --git a/README.md b/README.md
iindex 7967050..f55e5e0 100644
---- a/README.md
++++ b/README.md
@@@ -112,6 +112,9 @@ server/
object.pack (durable segments packed into ~60 s chunk objects; MEDIA_HLS_ENABLED),
rendition.create (an on-demand 720p H.264/AAC rendition of a media object into media_timeline;
MEDIA_HLS_ENABLED + MEDIA_RENDITIONS),
++ clip.cut (a clip's cut/re-cut; with MEDIA_HLS_ENABLED + MEDIA_MATERIALIZED_CLIPS and a source
++ timeline it materializes over the source's segments, only the edges re-encoded —
++ docs/materialized-clips.md),
vod.finalize (orphans and failed finalizes, with backoff),
vod.duration.reconcile (stored vs measured durations, local and B2/R2),
storage.orphans.scan (monthly storage orphan report, service-wide, report only),
@@@ -272,12 +275,12 @@ lossless `.master.mkv` recovery archive. A `.seekable` sidecar is remuxed every
| method | path | notes |
|---|---|---|
--| POST | `/clips` | `{ vod_id, start_s, end_s, title?, description?, user_id?, visibility?, auto_generated?, materialize? }` → over a VOD with a CMAF timeline (`MEDIA_HLS_ENABLED`): a **virtual** clip, **201** ready at once (the clip, `storage_provider: 'timeline'`), played as HLS over the source's segments (`/c/:id` redirects to its playlist); with `materialize: true` or no timeline → **202** `{ id, status: 'processing', job_id }`; cut (`clip.cut`) runs in background (from the local file or a presigned B2/R2 URL); duplicate windows are deduplicated; live recordings are clamped to flushed footage. Multipart `video` = direct upload of an already-cut blob → **201** ready |
--| GET | `/clips/:id` | status: `processing | ready | failed`; with `MEDIA_HLS_ENABLED` also `hls_url` (and `hls_expires_at` when signed) — the clip's playlist over its own timeline or, virtual, its source's window (docs/media-fabric.md §4) |
++| POST | `/clips` | `{ vod_id, start_s, end_s, title?, description?, user_id?, visibility?, auto_generated?, materialize? }` → over a VOD with a CMAF timeline (`MEDIA_HLS_ENABLED`): a **virtual** clip, **201** ready at once (the clip, `storage_provider: 'timeline'`), played as HLS over the source's segments (`/c/:id` redirects to its playlist); with `materialize: true` or no timeline → **202** `{ id, status: 'processing', job_id }`; cut (`clip.cut`) runs in background — with `MEDIA_HLS_ENABLED` + `MEDIA_MATERIALIZED_CLIPS=1` and a source timeline it is **materialized** over the source's bytes (the clip's own media_timeline rows copy the interior segments; only the two window edges are re-encoded, docs/materialized-clips.md), otherwise the full re-encode (from the local file or a presigned B2/R2 URL); duplicate windows are deduplicated; live recordings are clamped to flushed footage. Multipart `video` = direct upload of an already-cut blob → **201** ready |
++| GET | `/clips/:id` | status: `processing | ready | failed`; with `MEDIA_HLS_ENABLED` also `hls_url` (and `hls_expires_at` when signed) — the clip's playlist over its own timeline (a materialized clip's persisted rows) or, virtual, its source's window (docs/media-fabric.md §4) |
| POST | `/clips/:id/recut` | `{ materialize? }` → a file-less clip over a timeline turns virtual (200, the clip); otherwise (or `materialize: true`) **202** `{ id, status, job_id }` (`clip.cut`; a virtual clip stays ready while it is cut) |
| GET | `/clips?limit&offset&vod_id&stream_id&user_id&channel_user_id&hide_self&include_private&order&auto_generated&status&since` | list; `channel_user_id` = clipped-channel owner; `include_private` app-key only; `auto_generated=1\|0` = cut by the app's automation (AI auto-clips) or made by a person; `status=ready` = playable clips only; `since` as for VODs |
| PUT | `/clips/:id` | `{ title?, visibility?, auto_generated? }` (`auto_generated` app key only, 403 when acting for a user) |
--| DELETE | `/clips/:id` | local + offloaded objects + row |
++| DELETE | `/clips/:id` | local + offloaded objects + row; a timeline clip's own rows and edge bytes go with it, while every location the source still names — a materialized clip's interior — stays (docs/materialized-clips.md) |
### Pastes
@@@ -499,7 +502,7 @@ aggregated and stay queued. The runbook is `docs/cutover-media-billing-readings.
| `GET /live/:sel/transcript.json` | **transcript + AI timeline API** — full audio-transcription log and AI overview timeline. `:sel` = slot id / slot slug → slot-scoped sessions; **`@username`** → user-scoped (all their slots, works while offline). Returns `{ live, current, sessions[] (each: ai_overview, transcript, duration…), streamer (overview + stream memories), user }`. `?limit=1..50` sessions (default 10), `?app=`. Cached **30s** (that's the rate limit), CORS-open. |
| `GET /live/:sel/chat-insight.json` | **chat insight API** — a user's chat-related AI insight/timeline (`:sel` = `@username` or numeric user id): today-vs-alltime chat overviews, condensed memory, event timeline, plus their streamer overview + stream memories when they stream. Proxied from the app's public chat-AI API over loopback; cached **30s**, CORS-open. |
| `GET /v/:id/transcript.json` | **VOD transcript API** — transcript + AI overview for one existing VOD id (`{ vod_id, title, duration_seconds, ai_overview, transcript, ai_analyzed_at }`). Private VODs → 404. Cached 30s, CORS-open. |
--| `GET /o/:id/master.m3u8`, `/o/:id/source/index.m3u8`, `/o/:id/source/{init.mp4,NNNNNN.m4s}`, `/o/:id/<rendition>/…` | segment-native video (F3.1), only with `MEDIA_HLS_ENABLED=1` (off by default; off, these 404): HLS playlists written from the object's `media_timeline` rows (VOD, `#EXT-X-ENDLIST`) and its CMAF segments (Range; a segment packed by `object.pack` (F3.3) is a byte range of its chunk). The master playlist lists one variant per rendition whose rows exist — the `source` cut plus, with `MEDIA_RENDITIONS=1`, an on-demand 720p rung cut by `rendition.create` and queued lazily by this route and by `download?format=json` (F4 slice 1, docs/media-fabric.md §5). Same check as `GET /o/:id` (and, like it, no `Access-Control-Allow-Origin` on a private or sandbox answer): private and sandbox objects only with the object's `?exp&sig` (`/api/v2/:app/objects/:id/download?format=json` → `hls_url`, a playlist token valid `MEDIA_HLS_PLAYLIST_TTL_S` (6 h; 60 s to 12 h) that no other route accepts; a download signature works too), carried onto every URI; deleted 410. The `object.cmaf` job makes the timeline; a recording's finalize queues it, and `download?format=json` queues it for an object that has none yet; the v1 VOD and clip answers carry the same `hls_url` (and `hls_expires_at` when signed). `noindex` |
++| `GET /o/:id/master.m3u8`, `/o/:id/source/index.m3u8`, `/o/:id/source/{init.mp4,init-head.mp4,init-tail.mp4,NNNNNN.m4s}`, `/o/:id/<rendition>/…` | segment-native video (F3.1), only with `MEDIA_HLS_ENABLED=1` (off by default; off, these 404): HLS playlists written from the object's `media_timeline` rows (VOD, `#EXT-X-ENDLIST`) and its CMAF segments (Range; a segment packed by `object.pack` (F3.3) is a byte range of its chunk). The master playlist lists one variant per rendition whose rows exist — the `source` cut plus, with `MEDIA_RENDITIONS=1`, an on-demand 720p rung cut by `rendition.create` and queued lazily by this route and by `download?format=json` (F4 slice 1, docs/media-fabric.md §5). Same check as `GET /o/:id` (and, like it, no `Access-Control-Allow-Origin` on a private or sandbox answer): private and sandbox objects only with the object's `?exp&sig` (`/api/v2/:app/objects/:id/download?format=json` → `hls_url`, a playlist token valid `MEDIA_HLS_PLAYLIST_TTL_S` (6 h; 60 s to 12 h) that no other route accepts; a download signature works too), carried onto every URI; deleted 410. The `object.cmaf` job makes the timeline; a recording's finalize queues it, and `download?format=json` queues it for an object that has none yet; the v1 VOD and clip answers carry the same `hls_url` (and `hls_expires_at` when signed). A materialized clip (F3.6, `MEDIA_MATERIALIZED_CLIPS=1`) plays its own persisted rows: the playlist carries `#EXT-X-DISCONTINUITY` + a fresh `#EXT-X-MAP` where a re-encoded edge's `init-head.mp4`/`init-tail.mp4` replaces the source init (docs/materialized-clips.md). `noindex` |
| `GET /robots.txt` | crawler policy (`openvibe-shared/seo.robotsTxt`: AI and search crawlers named, `/api/`, `/auth/`, `/internal/`, `/o/`, `/metrics`, `/live/`, `/me` disallowed) and the sitemap |
| `GET /sitemap.xml` | the media index and the watch pages Media is the canonical home of: public, playable VODs and clips of apps other than Live (Live lists its own `/vod` and `/clip` pages), never AI clips, private, unlisted, sandbox, recording or failed items; thumbnails as image entries. Every listed page renders `index, follow` |
| `GET /llms.txt` | a map of the site for language-model crawlers |
ddiff --git a/STATUS.json b/STATUS.json
iindex 9c72064..c420af0 100644
---- a/STATUS.json
++++ b/STATUS.json
@@@ -43,7 +43,8 @@
"packing after finalize (F3.3, behind MEDIA_HLS_ENABLED): the object.pack job concatenates durable segments into ~60 s chunk objects verified by sha256, each media_timeline row naming its byte range; the segment route serves a packed segment as a ranged read of its chunk; the per-segment keys are deleted once the chunk is durable; segment providers come from the placement router; a finished object.cmaf queues object.pack by itself (dedupeActive: a rerun joins an active pack, a later cut queues a new one)",
"on-demand renditions (F4 slice 1, behind MEDIA_HLS_ENABLED + MEDIA_RENDITIONS): the rendition.create job transcodes one 720p H.264/AAC rung of a ready media object into media_timeline through the same content-addressed rows, placement-router durable provider and two-phase commit as object.cmaf, forcing the source timeline's segment boundaries as keyframes; a source already 720p or smaller is skipped and a held object is left alone. The master playlist lists a rendition only once its rows exist and queues the missing ones lazily from /o/:id/master.m3u8 and download?format=json (dedupeActive + an object/rendition idempotency key), and /o/:id/<rendition>/index.m3u8 plus /o/:id/<rendition>/<name> serve it",
"timeline sprites (F3.1, behind MEDIA_HLS_ENABLED): object.sprite cuts its sheet from the object's media_timeline rows — each sample time picks the row covering it and one frame is decoded from that row's bytes (this node's segment, or a packed row's byte range in its chunk), checked against the row's sha256 — with the same layout and metadata as the seek path, so a packed long VOD is never decoded end to end; a source without a timeline keeps one fast seek per frame",
-- "HLS discovery on the v1 shapes (F3.c, behind MEDIA_HLS_ENABLED): the VOD and clip answers (vodPublic/clipPublic — get, list, create, update) carry hls_url (and hls_expires_at when the object is private or sandbox), the same field and access rules as GET /o/:id/download?format=json, read through one shared discovery (server/objects/hls.js); an object without a timeline gets object.cmaf queued and the field appears once it ran, and a virtual clip plays its source's window without being cut a timeline of its own"
++ "HLS discovery on the v1 shapes (F3.c, behind MEDIA_HLS_ENABLED): the VOD and clip answers (vodPublic/clipPublic — get, list, create, update) carry hls_url (and hls_expires_at when the object is private or sandbox), the same field and access rules as GET /o/:id/download?format=json, read through one shared discovery (server/objects/hls.js); an object without a timeline gets object.cmaf queued and the field appears once it ran, and a virtual clip plays its source's window without being cut a timeline of its own",
++ "materialized clips (F3.6, behind MEDIA_HLS_ENABLED + MEDIA_MATERIALIZED_CLIPS, off by default): clip.cut over a source timeline writes the clip's own media_timeline rows — the interior copies the source's segments (packed chunk ranges included), only the two window edges are re-encoded into fMP4 with their own inits (init-head.mp4/init-tail.mp4, negative seq, EXT-X-MAP + EXT-X-DISCONTINUITY in the playlist) — inserting them in one transaction that locks the named source rows FOR SHARE and aborts with a retryable error when one is gone or any location column moved (a pack), so a racing delete either waits and keeps the bytes or aborts the insert; the projection keeps virtual: true and gains materialized: true (readiness, playableSql and the verify job treat it as a virtual clip), sprites decode each edge with its own init, deleting the clip removes its rows and edge bytes while the source's bytes stay, and download?format=mp4 answers the playlist for a timeline-only clip instead of queueing a remux that cannot read its input"
],
-- "updated": "2026-10-04"
++ "updated": "2026-10-05"
}
ddiff --git a/docs/materialized-clips.md b/docs/materialized-clips.md
iindex b3a2c94..0cd53a3 100644
---- a/docs/materialized-clips.md
++++ b/docs/materialized-clips.md
@@@ -1,6 +1,7 @@
# Materialized clips over the source's bytes (F3.6, design)
--**Status:** design, 2026-10-05 (plan T4, `docs/media-fabric.md` §4 "Materialized clip"). It replaces the full
++**Status:** implemented (behind `MEDIA_MATERIALIZED_CLIPS`, off by default), 2026-10-05 (plan T4,
++`docs/media-fabric.md` §4 "Materialized clip"). It replaces the full
re-encode of `clip.cut` (`server/vod/clip-cutter.js`, libvpx at about 3.3× realtime) with a clip whose interior
segments **name the source's bytes** and whose two edges are re-encoded. It depends on F3.4 (PR #35, merged
2026-10-05): any number of objects may name one location, a location's bytes are deleted only when no row of any
ddiff --git a/docs/media-fabric.md b/docs/media-fabric.md
iindex 621a1bc..5868ec2 100644
---- a/docs/media-fabric.md
++++ b/docs/media-fabric.md
@@@ -167,7 +167,19 @@ med_xyz/{captions/*.vtt, storyboard.webp+.vtt, waveform, metadata}
answers the object's `remux` variant (`object.remux`,
`-movflags +faststart` for an MP4 source; the container stays the source's) as a signed URL, or queues the remux
(`dedupeActive`) and answers `202 { job_id }` while it is missing.
--- **F3.5, shipped: virtual and materialized clips over the timeline (§4).**
++- **F3.5, shipped: virtual clips over the timeline (§4).**
++- **F3.6, shipped (behind `MEDIA_HLS_ENABLED` + `MEDIA_MATERIALIZED_CLIPS`, off by default): materialized clips over
++ the source's bytes (§4, docs/materialized-clips.md).** A clip whose source has a timeline gets its own persisted
++ `media_timeline` rows: the interior segments are the source's rows (same location, packed chunk range and sha256,
++ renumbered and relative to the clip), only the two window edges are re-encoded (H.264/AAC fMP4) with their own init
++ rows (`init-head.mp4`/`init-tail.mp4`, negative seq), and the playlist switches init with `#EXT-X-DISCONTINUITY` +
++ `#EXT-X-MAP`. The rows are inserted in one transaction that locks the named source rows `FOR SHARE` and aborts with a
++ retryable JobError when one is gone or any location column moved (a pack), so a racing source delete either waits and
++ keeps the bytes or aborts the insert; the clip's projection keeps `virtual: true` and gains `materialized: true`, so
++ readiness, `playableSql` and the verify job treat it as a virtual clip; deleting the clip removes its rows and edge
++ bytes while every location the source still names stays; sprites decode each edge with its own init; a source without
++ a timeline (or the flag off) keeps the full re-encode, and `…/download?format=mp4` answers the playlist for a
++ timeline-only clip until `object.remux` gains a timeline input branch.
- **F3.1 sprites, shipped: a sprite sheet from the timeline.** `object.sprite` (with `MEDIA_HLS_ENABLED`) cuts its
frames from the object's `media_timeline` rows instead of seeking the source: each sample time picks the row covering
it (`timeline.segmentAt`) and one frame is decoded from that row's bytes — this node's segment file, or a packed row's
@@@ -176,10 +188,8 @@ med_xyz/{captions/*.vtt, storyboard.webp+.vtt, waveform, metadata}
and metadata are the seek path's, so the player contract is unchanged. A source without a timeline, or with
`MEDIA_HLS_ENABLED` off, keeps the one-fast-seek-per-frame path; captions stay with AI's `media.analyze`.
**Still open in F3:** the growing live/DVR playlist written as OpenRe segments (F3.2), write-behind durability for
-- live segments and its upload-lag metric, a materialized clip that references the source's chunks (reference counts
-- shipped, F3.4; its own persisted rows and boundary-only re-encode are open: docs/materialized-clips.md), captions on
-- the timeline, and Live's player moving to HLS. Segment-bucket demand (F2.4) now records reads per timeline segment;
-- deciding placement per segment stays F5's. Renditions are F4.
++ live segments and its upload-lag metric, captions on the timeline, and Live's player moving to HLS. Segment-bucket
++ demand (F2.4) now records reads per timeline segment; deciding placement per segment stays F5's. Renditions are F4.
## 4. Clips reuse the source
@@@ -203,22 +213,24 @@ med_xyz/{captions/*.vtt, storyboard.webp+.vtt, waveform, metadata}
with the clip's playlist token when closed), a browser navigation too (no watch page for it yet); the clip answers
(`GET /api/v1/:app/clips/:id` and the list/create/update shapes) carry its `hls_url` (the clip's own token when
closed), and `playback_url` is unchanged. Its object's metadata carries `virtual: true`: readiness counts it
-- `bytes_verified`/`playable` without a location of its own, and the verify job skips it. **Materialized** = today's
-- `clip.cut` full
-- re-encode into the clip's own file: with `materialize: true` on `POST /clips` or `POST /clips/:id/recut`, or when
-- the source has no timeline (or the flag is off). A virtual clip keeps playing while it is materialized and stays
-- virtual if the cut fails; a clip with a file is never turned virtual (its recut stays a cut). The retry sweeper
++ `bytes_verified`/`playable` without a location of its own, and the verify job skips it. **Materialized** = a clip
++ that asked for it (`materialize: true` on `POST /clips` or `POST /clips/:id/recut`), or whose source has no timeline:
++ with `MEDIA_HLS_ENABLED` + `MEDIA_MATERIALIZED_CLIPS` on and a source timeline it gets its own persisted rows over
++ the source's segments — only the edges re-encoded (F3.6, docs/materialized-clips.md) — and otherwise it is
++ `clip.cut`'s full re-encode into the clip's own file. A virtual clip keeps playing while it is materialized and
++ stays virtual if the cut fails; a clip with a file is never turned virtual (its recut stays a cut). The retry sweeper
recovers a failed, file-less clip the way a recut does: virtual when its source now has a timeline. Holds still inherit
through `clip_of`.
--- **Materialized clip** (shared externally, downloaded, edited, popular): middle GOPs are **referenced**, not copied
-- (the new manifest points at the source's packed chunks); only the two boundary segments are re-encoded for
-- frame-accurate cuts. Segments and chunks are content-addressed (sha256), so the reference count of a location is just
-- the rows that name it (F3.4, shipped: `timeline.namedElsewhere`, no counter table; migration 0003 indexes it) and
-- deleting a VOD that a clip still references keeps the referenced chunks — they go when the last naming object is
-- removed, and holds still freeze everything. The clip's own persisted rows over the source's locations are the next
-- step (§3, F3.4 carries the deletion rule). Inserting a clip's rows over a source's locations must check, in the same
-- transaction, that a source row still names each location (and take a per-location lock with the delete check), or a
-- concurrent delete can drop bytes the new clip names.
++- **Materialized clip, shipped (F3.6; docs/materialized-clips.md)** (shared externally, downloaded, edited, popular):
++ middle GOPs are **referenced**, not copied (the clip's own persisted rows point at the source's locations, packed
++ chunk ranges included); only the two boundary segments are re-encoded for frame-accurate cuts. Segments and chunks
++ are content-addressed (sha256), so the reference count of a location is just the rows that name it (F3.4, shipped:
++ `timeline.namedElsewhere`, no counter table; migration 0003 indexes it) and deleting a VOD that a clip still
++ references keeps the referenced chunks — they go when the last naming object is removed, and holds still freeze
++ everything. The clip's rows are inserted in one transaction that locks the named source rows `FOR SHARE` and aborts
++ when one is gone or any location column moved (a pack), so a concurrent delete cannot drop bytes the new clip names.
++ The path is behind `MEDIA_MATERIALIZED_CLIPS` (off by default); a download still answers the playlist until
++ `object.remux` gains a timeline input branch.
## 5. Derivatives are cost-managed computed caches
@@@ -403,10 +415,10 @@ reads are metered by CDN logs. Nothing here bills delivery.
- **F3 segment-native video:** the timeline index and the CMAF/HLS source representation of finished video (F3.1,
shipped; §3), packing into ~60 s chunks (F3.3, shipped; §3), the timeline queued at finalize (and lazily on
download), signed playlists outliving a download URL and the faststart MP4 fallback (F3.c, shipped; §3), virtual and
-- materialized clips (F3.5, shipped; §4) and sprite sheets cut from the timeline's rows (F3.1, shipped; §3); still
++ materialized clips (F3.5/F3.6, shipped; §4) and sprite sheets cut from the timeline's rows (F3.1, shipped; §3); still
open: live DVR from OpenRe (CMAF/HLS recording with the growing
-- playlist), materialized clips referencing the source's chunks, captions on the
-- timeline (the demand rollups already record reads per timeline segment, F2.4); Live's player moves to HLS.
++ playlist), captions on the timeline (the demand rollups already record reads per timeline segment, F2.4); Live's
++ player moves to HLS.
- **F4 reactive derivatives:** on-demand renditions and image variants, keep-vs-regenerate economics, AV1 for viral
VODs, compute placement. (Slice 1 shipped: one on-demand 720p rung, queued by the master playlist / JSON download
and cut into the timeline; the economics, AV1 and compute placement are still open.)
ddiff --git a/migrations/0005_materialized_clips.sql b/migrations/0005_materialized_clips.sql
nnew file mode 100644
iindex 0000000..0715adc
---- /dev/null
++++ b/migrations/0005_materialized_clips.sql
@@@ -0,0 +1,13 @@
++-- phase: expand
++-- Materialized clips over the source's bytes (docs/materialized-clips.md, F3.6). A media segment row may name the init
++-- segment it decodes with (init_name; NULL = the rendition's seq 0 init, every existing row's meaning), and a clip's
++-- rendition carries the two re-encoded edge inits as rows with negative seq (-1 head, -2 tail). The primary key
++-- (object_id, rendition, seq) already distinguishes the two values; this migration replaces 0002's seq >= 0 CHECK with
++-- one that admits exactly the two edge init names. Additive only: an older release never reads init_name and never
++-- writes a negative seq.
++ALTER TABLE media_timeline ADD COLUMN IF NOT EXISTS init_name text COLLATE "C";
++-- 0002's inline column check is named media_timeline_seq_check (PostgreSQL's default for a column CHECK). DROP IF
++-- EXISTS + ADD together, so re-applying this migration to an already-migrated database is a no-op, never an error.
++ALTER TABLE media_timeline DROP CONSTRAINT IF EXISTS media_timeline_seq_check;
++ALTER TABLE media_timeline ADD CONSTRAINT media_timeline_seq_check
++ CHECK (seq >= 0 OR name IN ('init-head.mp4', 'init-tail.mp4'));
ddiff --git a/server/config.js b/server/config.js
iindex 2bb17b4..90a3505 100644
---- a/server/config.js
++++ b/server/config.js
@@@ -94,6 +94,10 @@ const config = {
// On-demand renditions (docs/media-fabric.md §5/F4): the rendition.create job and the extra master-playlist
// variants. Also needs `enabled`; off by default, so no new rendition is cut or queued (an existing one serves).
renditions: ['1', 'true', 'on'].includes(String(process.env.MEDIA_RENDITIONS || '').toLowerCase()),
++ // Materialized clips (docs/materialized-clips.md/F3.6): with `enabled` and a source that has a source timeline,
++ // clip.cut copies the source's interior segments into the clip's own rows and re-encodes only the two window
++ // edges, instead of the full re-encode. Off by default: every clip is cut exactly as before.
++ materialized: ['1', 'true', 'on'].includes(String(process.env.MEDIA_MATERIALIZED_CLIPS || '').toLowerCase()),
},
jobs: {
ddiff --git a/server/index.js b/server/index.js
iindex d8ed2ba..cac89bb 100644
---- a/server/index.js
++++ b/server/index.js
@@@ -284,6 +284,8 @@ const ready = (async () => {
// Object model: purge native objects whose soft-delete retention has passed (held ones are kept).
every(60 * 60 * 1000, async () => {
try { const n = await require('./objects/model').purgeExpired(); if (n) console.log(`[Objects] Purged ${n} expired deleted object(s)`); } catch (err) { console.warn('[Objects] purge:', err.message); }
++ // A projected object (a deleted clip's/VOD's) whose durable timeline delete failed keeps its rows; retry them.
++ try { const r = await require('./objects/timeline').sweepPendingRemovals(); if (r.objects) console.log(`[Objects] Timeline retry: ${r.removed} row(s) of ${r.objects} gone object(s) removed, ${r.pending} still pending`); } catch (err) { console.warn('[Objects] timeline retry:', err.message); }
// Incomplete multipart uploads past MEDIA_MULTIPART_TTL_HOURS: their parts are deleted (the objects stay uploading).
try { const r = await require('./objects/multipart').purgeExpired(); if (r.expired || r.orphan_dirs) console.log(`[Objects] Multipart: ${r.expired} expired session(s), ${r.orphan_dirs} orphan part dir(s) removed`); } catch (err) { console.warn('[Objects] multipart purge:', err.message); }
// Native objects' popularity (ADR-021): a finished UTC day's viewer hashes and salt are deleted, only its counts stay (30 days).
ddiff --git a/server/jobs/previews.js b/server/jobs/previews.js
iindex 5b869f2..0ed76ed 100644
---- a/server/jobs/previews.js
++++ b/server/jobs/previews.js
@@@ -236,23 +236,38 @@ function sweepPieces(dir) {
for (const n of names) if (/^seg-\d+\.mp4$/.test(n)) { try { fs.unlinkSync(path.join(dir, n)); } catch { /* gone */ } }
}
++/**
++ * The init row a media segment decodes with: the row its init_name names (a materialized clip's re-encoded edges have
++ * their own init), else the rendition's seq 0 init (a NULL init_name, every pre-F3.6 row). null = no init at all.
++ */
++function initRowFor(rows, row) {
++ const named = row && row.init_name ? rows.find((r) => Number(r.seq) <= 0 && r.name === row.init_name) : null;
++ return named || rows.find((r) => Number(r.seq) === 0) || null;
++}
++
/**
* The sheet from the timeline (F3.1): each sample time picks the row covering it (segmentAt) and one frame is
-- * decoded from that row's bytes — init.mp4 + the segment as one small file, never the source, never a full-file
-- * decode. Layout, metadata and the player contract are the seek path's.
++ * decoded from that row's bytes — its init segment + the segment as one small file, never the source, never a
++ * full-file decode. Layout, metadata and the player contract are the seek path's.
*/
async function runTimelineSprite(job, ctx, src, p, rows, segs) {
-- const initRow = rows.find((r) => Number(r.seq) === 0) || null;
// Pieces are deleted as the samples move past them, so only one or two are on disk at once.
-- const initLen = initRow ? Number(initRow.byte_length) || 0 : 0;
++ const maxInit = rows.filter((r) => Number(r.seq) <= 0).reduce((m, r) => Math.max(m, Number(r.byte_length) || 0), 0);
const maxSeg = segs.reduce((m, r) => Math.max(m, Number(r.byte_length) || 0), 0);
-- await d.checkRoom(src, 2 * (maxSeg + initLen) + 4 * MB);
++ await d.checkRoom(src, 2 * (maxSeg + maxInit) + 4 * MB);
const duration = await spriteDuration(src, segs);
if (!(duration > 0)) throw new JobError('no_duration', 'The timeline has no duration to spread frames over', { permanent: true });
const layout = spriteLayout(duration, p);
const dir = d.workDir(job.id);
-- // The init segment is read once and prepended to every piece; a media segment alone is not playable.
-- const initBody = initRow ? await rowBuffer(initRow, ctx) : null;
++ // Each init segment is read once and prepended to the pieces that decode with it; a media segment alone is not
++ // playable. A materialized clip's edges name their own init (initRowFor), the interior uses the seq 0 init.
++ const initBodies = new Map();
++ const initBodyFor = async (row) => {
++ const init = initRowFor(rows, row);
++ if (!init) return null;
++ if (!initBodies.has(init.name)) initBodies.set(init.name, await rowBuffer(init, ctx));
++ return initBodies.get(init.name);
++ };
const pieceFile = (seq) => path.join(dir, `seg-${String(seq).padStart(6, '0')}.mp4`);
const dropPiece = (seq) => { if (seq != null) { try { fs.unlinkSync(pieceFile(seq)); } catch { /* already gone */ } } };
sweepPieces(dir); // pieces a previous attempt left behind are rebuilt from the rows as needed
@@@ -264,6 +279,7 @@ async function runTimelineSprite(job, ctx, src, p, rows, segs) {
let file = null;
if (body) {
file = pieceFile(seq);
++ const initBody = await initBodyFor(row);
fs.writeFileSync(file, initBody && seq !== 0 ? Buffer.concat([initBody, body]) : body);
}
cache.set(seq, file);
@@@ -330,5 +346,6 @@ module.exports = {
waveform: { lane: 'heavy', maxAttempts: 3, timeoutMs: 3 * 3600 * 1000, needsObject: true, validate: validateWaveform, run: runWaveform },
sprite: { lane: 'heavy', maxAttempts: 3, timeoutMs: 3 * 3600 * 1000, needsObject: true, validate: validateSprite, run: runSprite },
spriteLayout,
++ initRowFor, // tested directly: each segment decodes with its own init (materialized clips, F3.6)
rowBuffer, // tested directly: the durable ranged-GET guards
};
ddiff --git a/server/objects/hls.js b/server/objects/hls.js
iindex a21b186..c45c55f 100644
---- a/server/objects/hls.js
++++ b/server/objects/hls.js
@@@ -131,7 +131,9 @@ async function discoveryForObject(obj) {
return null;
}
// The source timeline exists: queue any missing rendition (F4) so the master playlist can list it next time.
-- if (!tl.window) await queueRenditionsOnce(obj);
++ // A clip that plays without a file of its own (a virtual or materialized one) is never queued one — the job
++ // would fail with media_unavailable — so the lazy path leaves it alone, as queueRendition itself does.
++ if (!tl.window && !require('./timeline-queue').isVirtualClip(obj)) await queueRenditionsOnce(obj);
return playlistFor(obj, await isOpen(obj));
} catch (err) {
console.warn(`[HLS] Discovery failed for ${obj.id}:`, err.message);
ddiff --git a/server/objects/model.js b/server/objects/model.js
iindex bef363c..0390b78 100644
---- a/server/objects/model.js
++++ b/server/objects/model.js
@@@ -333,8 +333,11 @@ function clipProjection(row) {
const clipsDir = path.resolve(config.vod.clipsPath);
const { locs, canonical } = tieredLocations(row, [row.file_path || null, row.file_path ? path.join(clipsDir, path.basename(row.file_path)) : null]);
const status = row.status || 'ready';
-- // A virtual clip (storage_provider 'timeline', F3.5) has no file of its own: it plays its window of the source VOD's
-- // timeline (objects/routes.js timelineOf), so it is ready without one.
++ // A clip with no file and storage_provider 'timeline' (F3.5) has no location of its own: a virtual clip plays its
++ // window of the source VOD's timeline, a materialized clip (F3.6) its own persisted rows. Both keep `virtual: true`
++ // — readiness, playableSql and the verify job read it as "no location of its own needed" — and a materialized
++ // clip's metadata also carries `materialized: true`, written by clip.cut's ready transition in the same
++ // transaction and preserved here by project()'s metadata merge (a clips row has no column to derive it from).
const virtual = !row.file_path && row.storage_provider === 'timeline';
const lifecycle = status === 'processing' ? 'uploading' : status === 'failed' ? 'failed' : (row.file_path || virtual ? 'ready' : 'failed');
const local = locs.find(l => l.provider === 'local' && l.state === 'present');
@@@ -346,7 +349,7 @@ function clipProjection(row) {
size_bytes: local ? local.size_bytes : 0,
metadata: { title: row.title || null, duration_seconds: Number(row.duration_seconds) || 0, start_time: row.start_time, end_time: row.end_time,
channel_user_id: row.channel_user_id || null, auto_generated: !!row.auto_generated,
-- // Marks the clip virtual for readiness and the verify job; undefined drops the key once it is materialized.
++ // Marks the clip virtual for readiness and the verify job; undefined drops the key once it has a file.
virtual: virtual || undefined },
created_at: row.created_at, locations: locs, canonical,
};
ddiff --git a/server/objects/routes.js b/server/objects/routes.js
iindex 71ab312..d305663 100644
---- a/server/objects/routes.js
++++ b/server/objects/routes.js
@@@ -36,7 +36,8 @@
*
* Public bytes: GET /o/:id (publicRouter) — public/unlisted objects openly,
* private ones only with a valid ?exp&sig from /download. With MEDIA_HLS_ENABLED, the object's timeline as HLS
-- * under the same check: /o/:id/master.m3u8, /o/:id/source/index.m3u8, /o/:id/source/{init.mp4,NNNNNN.m4s}.
++ * under the same check: /o/:id/master.m3u8, /o/:id/source/index.m3u8,
++ * /o/:id/source/{init.mp4,init-head.mp4,init-tail.mp4,NNNNNN.m4s}.
*/
'use strict';
@@@ -692,6 +693,17 @@ router.get('/:id/download', read, limits('media.object.download'), async (req, r
*/
async function downloadMp4(req, res, obj) {
if (!require('../jobs/derive').isMediaObject(obj)) return problem(res, 400, 'media.object.not_media', 'Only video and audio objects have an MP4 download');
++ // A timeline-only clip (a virtual one, or a materialized clip whose bytes are its persisted media_timeline rows)
++ // has no file for a remux to read: answer its HLS playlist instead of queueing a job that cannot read its input
++ // (docs/materialized-clips.md §Downloads, the interim rule until object.remux gains a timeline input branch).
++ const md = model.parseJson(obj.metadata, {});
++ if (obj.kind === 'clip' && md.virtual === true) {
++ const info = await hls.discoveryForObject(obj);
++ if (info && info.hls_url) {
++ if (['1', 'true'].includes(String(req.query.redirect || ''))) return res.redirect(302, info.hls_url);
++ return res.json({ ...info, url: info.hls_url, expires_at: info.hls_expires_at || null, public: !info.hls_expires_at });
++ }
++ }
const variant = await model.getVariant(obj.id, 'remux');
const mp4 = variant && await model.getObject(variant.derived_object_id);
if (mp4 && mp4.lifecycle_status === 'ready') {
@@@ -780,7 +792,9 @@ const publicRouter = express.Router();
// Like GET /o/:id, a private or sandbox answer carries no Access-Control-Allow-Origin; a public one may be read cross-origin.
// A segment's copy is the placement router's choice; a packed one (F3.3) is a ranged read of its ~60 s chunk.
const HLS_TYPE = 'application/vnd.apple.mpegurl';
--const SEGMENT_NAME = /^(init\.mp4|\d{6,9}\.m4s)$/;
++// A rendition's seq 0 init plus the two edge inits of a materialized clip's source rendition (negative seq,
++// docs/materialized-clips.md F3.6); every init row (seq <= 0) is served as video/mp4, a media segment as iso.segment.
++const SEGMENT_NAME = /^(init\.mp4|init-head\.mp4|init-tail\.mp4|\d{6,9}\.m4s)$/;
const RENDITION_NAME = /^[a-z0-9_-]{1,32}$/;
const SLICE_FETCH_MS = 15000; // a ranged read of one packed segment from B2/R2
// clipWindow/timelineOf (a clip's window over its source, and the rows its playlists are written from) live in
@@@ -884,7 +898,7 @@ async function sendSlice(req, res, { file, url, offset, length, headers }) {
*/
async function sendSegmentRow(req, res, { obj, closed }, row, rendition) {
const timeline = require('./timeline');
-- const mime = Number(row.seq) === 0 ? 'video/mp4' : 'video/iso.segment';
++ const mime = Number(row.seq) <= 0 ? 'video/mp4' : 'video/iso.segment';
const headers = { 'Content-Type': mime, 'Cache-Control': closed ? 'private, no-store' : 'public, max-age=3600' };
const packed = !!row.packed_object_id;
const slice = { offset: Number(row.byte_offset), length: Number(row.byte_length), headers };
ddiff --git a/server/objects/timeline-queue.js b/server/objects/timeline-queue.js
iindex 36d9c91..5bbf8dd 100644
---- a/server/objects/timeline-queue.js
++++ b/server/objects/timeline-queue.js
@@@ -72,6 +72,15 @@ async function queuePack(appId, objectId, { rendition = null } = {}) {
*/
function renditionNames() { return require('../jobs/rendition').renditionNames(); }
++/**
++ * Whether an object is a clip that plays without a file of its own: a virtual clip (F3.5) or a materialized one
++ * (F3.6), both given metadata.virtual by clipProjection. rendition.create needs a file (resolveSource answers null
++ * without file_path), so neither one can ever be cut a rendition.
++ */
++function isVirtualClip(obj) {
++ return !!obj && obj.kind === 'clip' && require('./model').parseJson(obj.metadata, {}).virtual === true;
++}
++
/**
* Queue `rendition.create` for each rung the object does not have yet (one, '720p'), joined per object and rendition.
* `rendition` names a single rung instead of the whole ladder. → the first job id, or null when nothing was queued.
@@@ -83,8 +92,9 @@ async function queueRendition(appId, objectId, { rendition = null } = {}) {
// A held object is left alone (as the rendition job itself refuses it): queue nothing, so a hold does not burn jobs.
if (await require('./model').isHeld(obj.id)) return null;
const timeline = require('./timeline');
-- // A virtual clip (no timeline of its own) plays its source's window: it gets no rendition of its own.
-- if (obj.kind === 'clip' && !await timeline.has(obj.id)) return null;
++ // A clip that plays without a file of its own gets no rendition of its own: a virtual clip (no rows, F3.5) plays
++ // its source's window; a materialized one (own rows over the source's bytes, F3.6) has no file to transcode from.
++ if (isVirtualClip(obj) || (obj.kind === 'clip' && !await timeline.has(obj.id))) return null;
const queue = require('../jobs/queue');
const names = rendition ? [String(rendition)] : renditionNames();
const srcVer = sourceVersion(await timeline.list(obj.id, timeline.SOURCE));
@@@ -102,4 +112,4 @@ async function queueRendition(appId, objectId, { rendition = null } = {}) {
return first;
}
--module.exports = { queueCmaf, queuePack, queueRendition };
++module.exports = { queueCmaf, queuePack, queueRendition, isVirtualClip };
ddiff --git a/server/objects/timeline.js b/server/objects/timeline.js
iindex 4943ecb..4259082 100644
---- a/server/objects/timeline.js
++++ b/server/objects/timeline.js
@@@ -25,6 +25,12 @@
* object.pack re-keys every row that names a packed segment's old location (same sha256) to the chunk in the same
* transaction, so a clip follows its source into the chunk and the per-segment location is freed only once nothing names
* it. A source's own purge deletes its rows; the shared bytes go when the last naming object does.
++ *
++ * Materialized clips (docs/materialized-clips.md, F3.6): a clip's own rows copy the source's interior segments (same
++ * location and sha256, renumbered and relative to the clip) and name their edge inits as rows with negative seq
++ * (-1 init-head.mp4, -2 init-tail.mp4; init_name on a media row says which init it decodes with, NULL meaning the
++ * rendition's seq 0 init, as every pre-F3.6 row does). The rows are written by the pinning transaction in
++ * vod/clip-materialize.js, never by replace(); replace() still drops negative-seq rows a new plan does not carry.
*/
'use strict';
@@@ -35,7 +41,11 @@ const db = require('../db/database');
const SOURCE = 'source';
const INIT_NAME = 'init.mp4';
--const FIELDS = ['name', 'start_ms', 'end_ms', 'keyframe_ms', 'key', 'local_path', 'durable_provider', 'packed_object_id', 'byte_offset', 'byte_length', 'sha256', 'durability'];
++// The re-encoded edges of a materialized clip (docs/materialized-clips.md, F3.6): their own init segments, rows with
++// negative seq (-1 head, -2 tail) so they never collide with the media segments or the rendition's seq 0 init.
++const HEAD_INIT = 'init-head.mp4';
++const TAIL_INIT = 'init-tail.mp4';
++const FIELDS = ['name', 'start_ms', 'end_ms', 'keyframe_ms', 'key', 'local_path', 'durable_provider', 'packed_object_id', 'byte_offset', 'byte_length', 'sha256', 'durability', 'init_name'];
function localRoot() { return path.join(path.resolve(config.objects.path), '.timeline'); }
/** `version` names the immutable set of bytes (the sha's prefix): a re-cut lands beside the old one instead of on it. */
@@@ -114,8 +124,15 @@ async function replace(objectId, rendition, rows, { jobId = null, expect = null,
[objectId, rendition, r.seq, ...FIELDS.map((f) => r[f] ?? null), jobId]);
out[prev ? 'updated' : 'inserted']++;
}
-- const last = rows.reduce((m, r) => Math.max(m, r.seq), -1);
-- out.removed = (await db.run('DELETE FROM media_timeline WHERE object_id = ? AND rendition = ? AND seq > ?', [objectId, rendition, last])).changes || 0;
++ const last = rows.reduce((m, r) => Math.max(m, Number(r.seq)), -1);
++ // The negative-seq edge inits of a materialized clip are not "past the end": a row (positive or negative) not
++ // in the new set goes. A replace that carries no edge init drops the ones an earlier plan wrote.
++ const negatives = rows.map((r) => Number(r.seq)).filter((seq) => seq < 0);
++ const tail = negatives.length
++ ? `(seq > ? OR (seq < 0 AND seq NOT IN (${negatives.map(() => '?').join(', ')})))`
++ : '(seq > ? OR seq < 0)';
++ out.removed = (await db.run(`DELETE FROM media_timeline WHERE object_id = ? AND rendition = ? AND ${tail}`,
++ [objectId, rendition, last, ...negatives])).changes || 0;
if (dropOtherRenditions) {
dropped = await db.all('SELECT * FROM media_timeline WHERE object_id = ? AND rendition <> ?', [objectId, rendition]);
if (dropped.length) await db.run('DELETE FROM media_timeline WHERE object_id = ? AND rendition <> ?', [objectId, rendition]);
@@@ -223,6 +240,29 @@ async function removeObject(objectId) {
return { removed, pending };
}
++/**
++ * Retry the byte deletion of a projected object whose row is gone. removeObject reinserts a row whose durable copy
++ * could not be deleted (its `pending`), and purgeExpired (model.js, whose pending handling this mirrors: the rows are
++ * the record and a later pass retries) reaches only native objects. This is the pass that reaches a deleted clip's or
++ * VOD's leftover rows, and the hourly object sweep (server/index.js) calls it. Held objects are skipped, as every
++ * other delete path refuses them. → { objects, removed, pending }.
++ */
++async function sweepPendingRemovals({ limit = 200 } = {}) {
++ const rows = await db.all(`SELECT o.id FROM media_objects o
++ WHERE o.lifecycle_status = 'deleted' AND o.legacy_ref IS NOT NULL
++ AND EXISTS (SELECT 1 FROM media_timeline t WHERE t.object_id = o.id)
++ AND NOT EXISTS (SELECT 1 FROM clips c WHERE c.object_id = o.id)
++ AND NOT EXISTS (SELECT 1 FROM vods v WHERE v.object_id = o.id)
++ ORDER BY o.id LIMIT ?`, [Math.max(1, Number(limit) || 200)]);
++ let removed = 0, pending = 0;
++ for (const row of rows) {
++ if (await require('./model').isHeld(row.id)) continue;
++ const out = await removeObject(row.id);
++ removed += out.removed; pending += out.pending;
++ }
++ return { objects: rows.length, removed, pending };
++}
++
/**
* Point rows at the chunk they were packed into, in one transaction. `updates` = [{ row, key, local_path,
* durable_provider, packed_object_id, byte_offset }]; the location is the segment's old key (content-addressed, versioned
@@@ -297,15 +337,33 @@ const secs = (ms) => (Number(ms) / 1000).toFixed(3);
/** `?exp=…&sig=…` carried onto every URI of a signed playlist, so a private object's segments pass the same check. */
function withQuery(uri, query) { return query ? `${uri}?${query}` : uri; }
--/** The media playlist of a finished rendition: VOD, ENDLIST, one EXTINF per segment with its timeline duration. */
++/**
++ * The media playlist of a finished rendition: VOD, ENDLIST, one EXTINF per segment with its timeline duration. Each
++ * media segment decodes with the init its init_name names (a materialized clip's re-encoded edges), else the
++ * rendition's seq 0 init: EXT-X-MAP for the first segment, and #EXT-X-DISCONTINUITY + a fresh #EXT-X-MAP before any
++ * segment whose init differs from the previous segment's (docs/materialized-clips.md §Playlist). A playlist without
++ * edge inits comes out byte for byte as it did before.
++ */
function mediaPlaylist(rows, { query = '' } = {}) {
const segs = rows.filter((r) => Number(r.seq) > 0);
-- const init = rows.find((r) => Number(r.seq) === 0);
++ const baseInit = rows.find((r) => Number(r.seq) === 0) || null;
++ const namedInits = new Map(rows.filter((r) => Number(r.seq) < 0).map((r) => [r.name, r]));
++ const initOf = (r) => (r.init_name ? (namedInits.get(r.init_name) || { name: r.init_name }) : baseInit);
const target = Math.max(1, ...segs.map((r) => Math.ceil((Number(r.end_ms) - Number(r.start_ms)) / 1000)));
const lines = ['#EXTM3U', '#EXT-X-VERSION:7', `#EXT-X-TARGETDURATION:${target}`, `#EXT-X-MEDIA-SEQUENCE:${segs.length ? Number(segs[0].seq) : 1}`,
'#EXT-X-PLAYLIST-TYPE:VOD', '#EXT-X-INDEPENDENT-SEGMENTS'];
-- if (init) lines.push(`#EXT-X-MAP:URI="${withQuery(init.name, query)}"`);
-- for (const r of segs) lines.push(`#EXTINF:${secs(Number(r.end_ms) - Number(r.start_ms))},`, withQuery(r.name, query));
++ // A rendition with an init but no media row still emits its EXT-X-MAP (the pre-edge-init behaviour, byte for
++ // byte): the loop below would not reach it, so a rendition that produced an init and nothing else keeps its map.
++ if (!segs.length && baseInit) lines.push(`#EXT-X-MAP:URI="${withQuery(baseInit.name, query)}"`);
++ let prevInit = null;
++ for (const [i, r] of segs.entries()) {
++ const init = initOf(r);
++ const name = init ? init.name : null;
++ if (i === 0) { if (name) lines.push(`#EXT-X-MAP:URI="${withQuery(name, query)}"`); }
++ else if (name && name !== prevInit) lines.push('#EXT-X-DISCONTINUITY', `#EXT-X-MAP:URI="${withQuery(name, query)}"`);
++ lines.push(`#EXTINF:${secs(Number(r.end_ms) - Number(r.start_ms))},`, withQuery(r.name, query));
++ prevInit = name;
++ }
lines.push('#EXT-X-ENDLIST', '');
return lines.join('\n');
}
@@@ -341,8 +399,14 @@ function masterPlaylist(renditions, { query = '' } = {}) {
* local_path, durable_provider, packed_object_id and byte_* stay the source's, so a segment serves from the source's
* bytes (or its packed chunk), and its name, the playlist URI, never changes. A segment straddling an edge is served
* whole: a virtual clip's edges are the source's segment boundaries (a materialized clip cuts exactly).
++ *
++ * A materialized clip's rows carry negative-seq edge inits and per-segment init_name: a virtual clip over it would
++ * decode its edges with the wrong init, so building one is refused (the simplest correct answer, §Schema of
++ * docs/materialized-clips.md). Rows made only of copies of source rows (a clip whose window sits on boundaries) carry
++ * no edge init and still work.
*/
function clipRows(rows, startMs, endMs) {
++ if (rows.some((r) => Number(r.seq) < 0 || (r.init_name && r.init_name !== INIT_NAME))) return [];
const init = rows.find((r) => Number(r.seq) === 0);
const segs = rows
.filter((r) => Number(r.seq) > 0 && Number(r.end_ms) > startMs && Number(r.start_ms) < endMs)
@@@ -353,7 +417,7 @@ function clipRows(rows, startMs, endMs) {
}
module.exports = {
-- SOURCE, INIT_NAME, FIELDS, localRoot, localPathFor, keyFor, segmentName,
-- list, segments, byName, has, segmentAt, replace, TimelineChanged, isNamed, namedElsewhere, deleteBytes, removeObject, markPacked, durableProvider, locationsOf,
++ SOURCE, INIT_NAME, HEAD_INIT, TAIL_INIT, FIELDS, localRoot, localPathFor, keyFor, segmentName,
++ list, segments, byName, has, segmentAt, replace, TimelineChanged, isNamed, namedElsewhere, deleteBytes, removeObject, sweepPendingRemovals, markPacked, durableProvider, locationsOf,
mediaPlaylist, masterPlaylist, clipRows,
};
ddiff --git a/server/vod/clip-jobs.js b/server/vod/clip-jobs.js
iindex a61a540..d20c57b 100644
---- a/server/vod/clip-jobs.js
++++ b/server/vod/clip-jobs.js
@@@ -13,6 +13,9 @@
* file; ready at once (storage_provider 'timeline') and played as its window of
* the source's segments. clip.cut is the materialized path (asked for, or no
* timeline); materializing a virtual clip keeps it playing until the file is in.
++ * materialize (F3.6) with MEDIA_HLS_ENABLED + MEDIA_MATERIALIZED_CLIPS on and a source timeline, the
++ * cut keeps the source's interior segments (copied rows) and re-encodes only the
++ * two edges (vod/clip-materialize.js); the full re-encode is unchanged otherwise.
* start() sweeper: every 3 min, failed clips that still have
* attempts left are re-cut (or turned virtual, as a recut is,
* when the source now has a timeline). From the second attempt on, a
@@@ -57,7 +60,7 @@ async function recutClip(clipId, opts = {}) {
return result;
}
--async function _recutClip(clipId, { reason = 'recut', viaJob = false } = {}) {
++async function _recutClip(clipId, { reason = 'recut', viaJob = false, job = null, ctx = null } = {}) {
const clip = await db.getClipById(clipId);
if (!clip) return { ok: false, error: 'Clip not found' };
if (!clip.vod_id) return { ok: false, error: 'Clip has no source VOD' };
@@@ -71,6 +74,27 @@ async function _recutClip(clipId, { reason = 'recut', viaJob = false } = {}) {
if (isVirtual(clip)) await db.run('UPDATE clips SET cut_attempts = ?, cut_next_at = NULL WHERE id = ?', [attempt, clipId]);
else await objects().withObject('clip', clipId, async () => await db.run("UPDATE clips SET status = 'processing', cut_attempts = ?, cut_next_at = NULL WHERE id = ?", [attempt, clipId]));
++ const startTime = Number(clip.start_time) || 0;
++ const duration = Math.max(1, (Number(clip.end_time) || 0) - startTime);
++
++ // Materialized clips (docs/materialized-clips.md, F3.6): with MEDIA_HLS_ENABLED + MEDIA_MATERIALIZED_CLIPS and a
++ // source that has a source-rendition timeline, the clip's interior rows reference the source's segments and only
++ // its two edges are re-encoded — the VOD's own file is not needed. materialize() answers null when the path does
++ // not apply (flag off, no timeline, an empty window, not under a job), and the full re-encode below runs exactly
++ // as before. A failed or racing attempt goes through fail() like a failed cut: the job retries, and a virtual
++ // clip being materialized keeps playing.
++ if (job && config.hls.enabled && config.hls.materialized) {
++ let mat = null;
++ try { mat = await require('./clip-materialize').materialize({ job, ctx, clip, vod }); }
++ catch (err) { return await fail(clipId, clip, attempt, err.message, { viaJob, hopeless: !!err.permanent }); }
++ if (mat && mat.ok) {
++ await markMaterialized(clip, { duration: mat.duration, startTime, endMs: mat.end_ms });
++ console.log(`[Clips] Clip ${clipId} materialized over vod ${clip.vod_id}'s timeline (${startTime.toFixed(1)}-${(startTime + mat.duration).toFixed(1)}s, attempt ${attempt})`);
++ await require('../indexnow-notify').pingWatch('clip', await db.getClipById(clipId));
++ return { ok: true, duration_seconds: mat.duration };
++ }
++ }
++
// Attempt 2+: bring a cloud-stored VOD home first when the disk can take it.
let source = await vodStorage.resolveMediaSource(vod);
if (source && source.kind === 'url' && attempt >= 2) {
@@@ -86,31 +110,64 @@ async function _recutClip(clipId, { reason = 'recut', viaJob = false } = {}) {
}
if (!source) return await fail(clipId, clip, attempt, 'VOD media unavailable (not on disk and no cloud copy)', { viaJob });
-- const startTime = Number(clip.start_time) || 0;
-- const duration = Math.max(1, (Number(clip.end_time) || 0) - startTime);
const cut = await cutter.cutClipFile({ source: source.value, startTime, duration });
if (!cut.ok) return await fail(clipId, clip, attempt, cut.error, { viaJob });
// Thumbnail first (the clip.ready payload carries it), then the ready transition, the clip's
// object and its event in one transaction (webhooks.announce), then the webhook.
try { await require('../thumbnails/thumbnail-service').generateClipThumbnail(clipId, cut.filePath); } catch { /* */ }
await announce(clip.app_id, 'clip.ready', {
-- change: async () => await objects().withObject('clip', clipId, async () => await db.run(`UPDATE clips SET file_path = ?, duration_seconds = ?, end_time = ?, status = 'ready', cut_error = NULL, cut_next_at = NULL,
-- storage_provider = CASE WHEN storage_provider = 'timeline' THEN 'local' ELSE storage_provider END WHERE id = ?`,
-- [cut.filePath, cut.duration, startTime + cut.duration, clipId])),
++ change: async () => {
++ await objects().withObject('clip', clipId, async () => await db.run(`UPDATE clips SET file_path = ?, duration_seconds = ?, end_time = ?, status = 'ready', cut_error = NULL, cut_next_at = NULL,
++ storage_provider = CASE WHEN storage_provider = 'timeline' THEN 'local' ELSE storage_provider END WHERE id = ?`,
++ [cut.filePath, cut.duration, startTime + cut.duration, clipId]));
++ // A clip that had been materialized and now has a file of its own is no longer timeline-backed: project()
++ // would otherwise preserve metadata.materialized (the clips row has no column to derive it from).
++ await db.run(`UPDATE media_objects SET metadata = (COALESCE(NULLIF(metadata, ''), '{}')::jsonb - 'materialized')::text,
++ updated_at = ov_now() WHERE id = ?`, [clip.object_id]);
++ },
payload: async () => await _clipPublic(await db.getClipById(clipId)),
});
++ // The file supersedes any rows a previous materialization left. removeObject drops the clip's rows after the
++ // commit and deletes only the edge locations no other object's rows still name (F3.4) — the source's interior
++ // bytes stay. No rows is a no-op, so the ordinary file cut is unchanged.
++ const tl = await require('../objects/timeline').removeObject(clip.object_id)
++ .catch(err => { console.warn(`[Clips] Clip ${clipId} timeline cleanup failed: ${err.message}`); return null; });
++ if (tl && tl.pending) console.warn(`[Clips] Clip ${clipId}: ${tl.pending} timeline segment row(s) kept — their durable copy could not be deleted; the object sweep retries`);
console.log(`[Clips] Clip ${clipId} ${reason} OK from vod ${clip.vod_id} (${startTime.toFixed(1)}-${(startTime + cut.duration).toFixed(1)}s, attempt ${attempt})`);
// A public, person-made clip that just became ready is indexable: announce its watch page and the sitemap.
await require('../indexnow-notify').pingWatch('clip', await db.getClipById(clipId));
return { ok: true, duration_seconds: cut.duration };
}
++/**
++ * The materialized ready transition (docs/materialized-clips.md step 4): row, object and clip.ready in one
++ * transaction. file_path stays NULL and storage_provider stays 'timeline' (it has no location of its own); the
++ * object's metadata gains materialized: true, and clipProjection keeps virtual: true beside it, so readiness,
++ * playableSql and the verify job treat it as they treat a virtual clip.
++ */
++async function markMaterialized(clip, { duration, startTime, endMs }) {
++ await announce(clip.app_id, 'clip.ready', {
++ change: async () => await objects().withObject('clip', clip.id, async () => {
++ // end_time is the planned (snapped) end in source seconds, not startTime + duration: an edge that snapped
++ // back to its segment boundary, or a window clamped to the first segment, made originMs < startTime, so the
++ // old sum overshot the end and every re-cut started from the drifted value.
++ await db.run(`UPDATE clips SET file_path = NULL, storage_provider = 'timeline', duration_seconds = ?, end_time = ?,
++ status = 'ready', cut_error = NULL, cut_next_at = NULL WHERE id = ?`, [duration, endMs / 1000, clip.id]);
++ await db.run(`UPDATE media_objects SET metadata = jsonb_set(COALESCE(NULLIF(metadata, ''), '{}')::jsonb, '{materialized}', 'true'::jsonb)::text,
++ updated_at = ov_now() WHERE id = ?`, [clip.object_id]);
++ }),
++ payload: async () => await _clipPublic(await db.getClipById(clip.id)),
++ });
++}
++
/** A failed attempt. Under a job the queue schedules the retry, so nothing is left for the sweeper. */
--async function fail(clipId, clip, attempt, error, { viaJob = false } = {}) {
++async function fail(clipId, clip, attempt, error, { viaJob = false, hopeless: hopelessOverride = null } = {}) {
const msg = String(error || 'cut failed').slice(0, 500);
// Nothing to retry when the source itself has no footage there (empty recording, window
-- // past the end of what was captured): give up now instead of burning the whole ladder.
-- const hopeless = /No decodable footage|Error opening input: End of file|Source VOD no longer exists|Invalid data found when processing input/i.test(msg);
++ // past the end of what was captured): give up now instead of burning the whole ladder. A materialize
++ // error that was permanent (codec_unsupported) is passed through as hopeless the same way.
++ const hopeless = hopelessOverride != null ? hopelessOverride
++ : /No decodable footage|Error opening input: End of file|Source VOD no longer exists|Invalid data found when processing input/i.test(msg);
const more = !viaJob && attempt < MAX_ATTEMPTS && !hopeless;
const mins = BACKOFF_MIN[Math.min(attempt - 1, BACKOFF_MIN.length - 1)];
const nextAt = more ? new Date(Date.now() + mins * 60000).toISOString().replace('T', ' ').slice(0, 19) : null;
@@@ -238,11 +295,11 @@ const spec = {
if (!Number.isInteger(id) || id < 1) throw new (require('../jobs/queue').JobError)('media.job.invalid', 'clip.cut needs params.clip_id', { permanent: true });
return { clip_id: id, reason: String((params && params.reason) || 'cut').slice(0, 40) };
},
-- async run(job) {
++ async run(job, ctx) {
const { JobError } = require('../jobs/queue');
const clip = await db.getClipById(job.params.clip_id);
if (!clip || clip.app_id !== job.app_id) throw new JobError('not_found', 'The clip no longer exists', { permanent: true });
-- const r = await recutClip(clip.id, { reason: job.params.reason || 'cut', viaJob: true });
++ const r = await recutClip(clip.id, { reason: job.params.reason || 'cut', viaJob: true, job, ctx });
if (!r.ok) throw new JobError('clip_cut_failed', r.error || 'cut failed', { permanent: !!r.hopeless });
return { clip_id: clip.id, status: 'ready', duration_seconds: r.duration_seconds ?? null };
},
ddiff --git a/server/vod/clip-materialize.js b/server/vod/clip-materialize.js
nnew file mode 100644
iindex 0000000..5d90ccf
---- /dev/null
++++ b/server/vod/clip-materialize.js
@@@ -0,0 +1,316 @@
++/**
++ * OpenVibe.Media — materialized clips over the source's bytes (docs/materialized-clips.md, F3.6)
++ *
++ * A clip whose source has a CMAF timeline (MEDIA_HLS_ENABLED) can be materialized instead of re-encoded end to end:
++ * its interior segments are the source's own rows — same key/local_path/durable_provider/packed_object_id/byte_* and
++ * sha256, renumbered and relative to the clip — and only the two window edges are re-encoded from the source
++ * segment's bytes (its init + segment, read the way the sprite job reads them: local copy, packed slice, or a ranged
++ * durable GET, sha256 verified) into one fMP4 segment plus its own init, starting on a keyframe. The clip's rows land
++ * in ONE transaction that first locks the source rows it names FOR SHARE and aborts (retryable) when any is gone or any
++ * location column moved under it — the whole location, so a pack between the read and the pin aborts and re-plans.
++ *
++ * planWindow(rows, startMs, endMs) pure: head/interior/tail over the source's media segments. An edge within
++ * EDGE_MS (100 ms) of a segment boundary is not re-encoded: the clip starts or
++ * ends on that boundary. null = nothing playable in the window.
++ * pinRows({…}) the FOR SHARE read + timeline.replace in one transaction; throws JobError
++ * (media.timeline.changed, retryable) when a named source row is gone or moved.
++ * materialize({ job, ctx, clip, vod }) the whole path (encode edges, place their bytes content-addressed under the
++ * clip's own keys, pin the rows). null = not applicable (the flag off, no source
++ * timeline, an empty window): clip.cut keeps its full re-encode.
++ *
++ * Placement is publishRendition's (server/jobs/segments.js): bytes are staged until the pin commits and removed when
++ * it does not; a durable upload that fails is retried with the row kept local. Edge names: the two inits are the
++ * rows with negative seq (-1 init-head.mp4, -2 init-tail.mp4) the migration admits; every media piece (a copied
++ * interior segment or a re-encoded edge) is named by its clip seq, so two pieces never share a playlist URI.
++ */
++'use strict';
++
++const fs = require('fs');
++const path = require('path');
++const config = require('../config');
++const db = require('../db/database');
++const timeline = require('../objects/timeline');
++const model = require('../objects/model');
++const derive = require('../jobs/derive');
++const previews = require('../jobs/previews');
++const segments = require('../jobs/segments');
++const { JobError } = require('../jobs/queue');
++
++const EDGE_MS = 100;
++// The whole location of a source row the plan names: every column a pack or a re-cut could move, as
++// timeline.replace's expect compares (docs/materialized-clips.md step 3), plus byte_offset/byte_length.
++const LOCATION_FIELDS = ['key', 'local_path', 'durable_provider', 'packed_object_id', 'byte_offset', 'byte_length', 'sha256'];
++
++function locationOf(row) {
++ return LOCATION_FIELDS.map((f) => (row[f] == null ? '\u0000' : String(row[f]))).join('\n');
++}
++
++/**
++ * Plan a clip window over one rendition's rows.
++ * → { originMs, endMs, durationMs, sourceInit, sourceRows, pieces } or null.
++ * A piece is { kind: 'copy', source } or { kind: 'head'|'tail', source, start_ms, end_ms, init } (source milliseconds).
++ * sourceRows are the source rows the plan copies and therefore names: the seq 0 init when an interior copy needs it
++ * (init_name NULL rows decode with it), plus every copied segment. The pin compares exactly these.
++ */
++function planWindow(rows, startMs, endMs) {
++ const segs = rows.filter((r) => Number(r.seq) > 0).sort((a, b) => Number(a.seq) - Number(b.seq));
++ if (!segs.length) return null;
++ const first = segs[0];
++ const last = segs[segs.length - 1];
++ let start = Math.max(0, Math.floor(Number(startMs) || 0));
++ let end = Math.floor(Number(endMs) || 0);
++ if (!(end > start)) return null;
++ if (end <= Number(first.start_ms) || start >= Number(last.end_ms)) return null;
++ start = Math.max(start, Number(first.start_ms));
++ end = Math.min(end, Number(last.end_ms));
++ const at = (t) => segs.find((r) => Number(r.start_ms) <= t && t < Number(r.end_ms)) || null;
++ const headSeg = at(start);
++ const tailSeg = at(end - 1);
++ const overlapping = segs.filter((r) => Number(r.end_ms) > start && Number(r.start_ms) < end);
++ const pieces = [];
++ for (const r of overlapping) {
++ const rs = Number(r.start_ms);
++ const re = Number(r.end_ms);
++ // An edge within 100 ms of a segment boundary is not re-encoded: the clip starts or ends on that boundary.
++ const headCut = r === headSeg && start - rs > EDGE_MS;
++ const tailCut = r === tailSeg && re - end > EDGE_MS;
++ if (!headCut && !tailCut) { pieces.push({ kind: 'copy', source: r }); continue; }
++ const a = headCut ? start : rs;
++ const b = tailCut ? end : re;
++ if (!(b > a)) { pieces.push({ kind: 'copy', source: r }); continue; }
++ // One segment holding both edges is re-encoded once, as the head edge (its init-head carries it).
++ pieces.push({ kind: headCut ? 'head' : 'tail', source: r, start_ms: a, end_ms: b, init: headCut ? timeline.HEAD_INIT : timeline.TAIL_INIT });
++ }
++ if (!pieces.length) return null;
++ const firstPiece = pieces[0];
++ const lastPiece = pieces[pieces.length - 1];
++ const originMs = firstPiece.kind === 'copy' ? Number(firstPiece.source.start_ms) : firstPiece.start_ms;
++ const snapEndMs = lastPiece.kind === 'copy' ? Number(lastPiece.source.end_ms) : lastPiece.end_ms;
++ if (!(snapEndMs > originMs)) return null;
++ const sourceInit = rows.find((r) => Number(r.seq) === 0) || null;
++ const sourceRows = [];
++ if (pieces.some((p) => p.kind === 'copy') && sourceInit) sourceRows.push(sourceInit);
++ for (const p of pieces) if (p.kind === 'copy') sourceRows.push(p.source);
++ return { originMs, endMs: snapEndMs, durationMs: snapEndMs - originMs, sourceInit, sourceRows, pieces };
++}
++
++/** A copied source segment row as the clip's own: the location, bytes and times, seq renumbered, times relative. */
++function copiedRow(source, seq, originMs) {
++ const row = { seq };
++ for (const f of timeline.FIELDS) row[f] = source[f] == null ? null : source[f];
++ row.name = timeline.segmentName(seq); // the clip's own URI: a copied source name could collide with an edge's
++ row.start_ms = Number(source.start_ms) - originMs;
++ row.end_ms = Number(source.end_ms) - originMs;
++ row.keyframe_ms = source.keyframe_ms == null ? null : Number(source.keyframe_ms) - originMs;
++ row.init_name = null; // decodes with the rendition's seq 0 init
++ return row;
++}
++
++/** The source's seq 0 init as the clip's own row (seq 0, times 0), or null. */
++function copiedInitRow(source) {
++ const row = copiedRow(source, 0, 0);
++ row.name = timeline.INIT_NAME;
++ row.start_ms = 0;
++ row.end_ms = 0;
++ row.keyframe_ms = null;
++ return row;
++}
++
++/** A source segment's bytes with its init prepended, checked against the row's sha256 (previews.rowBuffer). */
++async function readPiece(segRow, initRow, ctx) {
++ const seg = await previews.rowBuffer(segRow, ctx);
++ if (!seg) throw new JobError('segment_unreadable', `${segRow.name}: no copy with the indexed sha256`, { retryAfterS: 1800 });
++ const init = initRow ? await previews.rowBuffer(initRow, ctx) : null;
++ if (initRow && !init) throw new JobError('segment_unreadable', `${initRow.name}: no copy with the indexed sha256`, { retryAfterS: 1800 });
++ return init ? Buffer.concat([init, seg]) : seg;
++}
++
++/** Re-encode one edge piece from the source segment's bytes into one fMP4 segment plus its own init. */
++async function encodeEdge({ ctx, piece, initRow, dir }) {
++ fs.mkdirSync(dir, { recursive: true });
++ const pieceFile = path.join(dir, 'in.mp4');
++ fs.writeFileSync(pieceFile, await readPiece(piece.source, initRow, ctx));
++ const relMs = piece.start_ms - Number(piece.source.start_ms);
++ const durMs = piece.end_ms - piece.start_ms;
++ const out = path.join(dir, 'out');
++ fs.mkdirSync(out, { recursive: true });
++ const budget = Math.min(30 * 60 * 1000, Math.max(180000, derive.budgetMs(Number(piece.source.byte_length) || 0) * 4));
++ const r = await derive.ffmpeg(['-y', '-nostdin', '-v', 'error',
++ ...(relMs > 0 ? ['-ss', (relMs / 1000).toFixed(3)] : []),
++ '-i', pieceFile, '-t', (durMs / 1000).toFixed(3),
++ '-map', '0:v:0?', '-map', '0:a:0?',
++ '-c:v', 'libx264', '-preset', 'veryfast', '-crf', '21', '-profile:v', 'main', '-pix_fmt', 'yuv420p',
++ '-c:a', 'aac', '-b:a', '128k', '-ac', '2', '-map_metadata', '-1',
++ '-f', 'hls', '-hls_time', '3600', '-hls_list_size', '0', '-hls_playlist_type', 'vod', '-hls_segment_type', 'fmp4',
++ '-hls_fmp4_init_filename', timeline.INIT_NAME,
++ '-start_number', '1', '-hls_segment_filename', path.join(out, '%06d.m4s'), path.join(out, 'index.m3u8')],
++ { signal: ctx.signal, timeoutMs: budget });
++ segments.aborted(ctx);
++ const tail = String(r.stderr).trim().split('\n').slice(-2).join(' ').slice(0, 300);
++ if (!r.ok) {
++ const unsupported = /Unknown encoder|not currently supported in container|could not find tag for codec|codec not supported/i.test(r.stderr);
++ throw new JobError(unsupported ? 'codec_unsupported' : 'ffmpeg_failed', `ffmpeg exited ${r.code}: ${tail}`, { permanent: unsupported });
++ }
++ const list = derive.existing(path.join(out, 'index.m3u8')) ? segments.parsePlaylist(fs.readFileSync(path.join(out, 'index.m3u8'), 'utf8')) : [];
++ const initFile = path.join(out, timeline.INIT_NAME);
++ if (list.length !== 1 || !derive.existing(initFile) || !derive.existing(path.join(out, list[0].name))) {
++ throw new JobError('ffmpeg_failed', `ffmpeg wrote ${list.length} segment(s), expected one edge segment: ${tail}`);
++ }
++ return { initFile, segFile: path.join(out, list[0].name) };
++}
++
++/**
++ * Place one produced file under the clip's own content-addressed location, as publishRendition does: a previous row
++ * with the same sha256 and a live location is reused (a packed one as it is), else the file is written to this node's
++ * timeline and uploaded to the router's durable provider. A failed upload is recorded, not thrown: the row stays
++ * local and the caller retries. Staged paths/keys are collected so a failed run can remove them.
++ */
++async function placeFile({ clipObj, prev, seq, name, file, contentType, target, remoteConfigured, stagedLocal, stagedRemote, failed, ctx }) {
++ const vodStorage = require('./vod-storage');
++ const size = fs.statSync(file).size;
++ const sha = await derive.sha256File(file);
++ const old = prev.get(Number(seq));
++ const match = !!old && old.sha256 === sha;
++ if (match && old.packed_object_id) {
++ // object.pack moved these bytes into a chunk: the same bytes, so the row and its chunk stay as they are.
++ return { key: old.key, local_path: old.local_path, durable_provider: old.durable_provider, packed_object_id: old.packed_object_id,
++ byte_offset: old.byte_offset, byte_length: old.byte_length, sha256: sha, durability: old.durability };
++ }
++ const version = sha.slice(0, 12);
++ const dest = timeline.localPathFor(clipObj, timeline.SOURCE, name, version);
++ const key = timeline.keyFor(clipObj.id, timeline.SOURCE, name, version);
++ let localPath = dest;
++ let durable = match && old.durability === 'durable' ? old.durable_provider : null;
++ if (match && old.local_path && derive.existing(old.local_path)) {
++ localPath = old.local_path; // the previous copy is exactly these bytes: keep it and its location
++ } else {
++ fs.mkdirSync(path.dirname(dest), { recursive: true });
++ fs.renameSync(file, dest);
++ stagedLocal.push(dest);
++ }
++ if (!durable && !target && remoteConfigured) failed.push(`${name}: no healthy durable provider`);
++ if (!durable && target) {
++ try {
++ if (!vodStorage.providerAvailable(target)) throw new Error(`${target} is unavailable`);
++ await vodStorage.uploadFile(target, key, localPath, contentType, { signal: ctx.signal });
++ stagedRemote.push({ provider: target, key });
++ durable = target;
++ } catch (err) {
++ if (ctx.signal.aborted) throw ctx.signal.reason || err; // a cancelled upload is not "not durable yet"
++ failed.push(`${name}: ${err.message}`);
++ }
++ }
++ return { key, local_path: localPath, durable_provider: durable, packed_object_id: null, byte_offset: null, byte_length: size, sha256: sha, durability: durable ? 'durable' : 'local' };
++}
++
++/**
++ * Insert the clip's rows in ONE transaction that first locks the named source rows FOR SHARE and aborts with a
++ * retryable JobError when any is gone or any location column differs from the planned row. The lock makes a source
++ * removeObject's row DELETE wait for this commit (its post-commit isNamed check then keeps the bytes); a removal or a
++ * pack that committed first fails the comparison and the job re-plans. `rows` is the clip's complete row set
++ * (negative-seq edge inits included); timeline.replace makes the clip's rendition exactly those rows.
++ */
++async function pinRows({ clipObjectId, sourceObjectId, rows, sourceRows = [], jobId = null }) {
++ await db.getDb().tx(async () => {
++ if (sourceRows.length) {
++ const seqs = sourceRows.map((r) => Number(r.seq));
++ // ORDER BY seq: the same order the packer and a source removal lock a source's rows in, so a pin cannot
++ // deadlock with them on the rows it takes FOR SHARE (a deadlock is retried twice, then fails the job).
++ const got = await db.all(`SELECT * FROM media_timeline WHERE object_id = ? AND rendition = ? AND seq IN (${seqs.map(() => '?').join(', ')}) ORDER BY seq FOR SHARE`,
++ [sourceObjectId, timeline.SOURCE, ...seqs]);
++ const bySeq = new Map(got.map((r) => [Number(r.seq), r]));
++ for (const want of sourceRows) {
++ const now = bySeq.get(Number(want.seq));
++ if (!now || locationOf(now) !== locationOf(want)) {
++ throw new JobError('media.timeline.changed', `the source timeline changed while materializing (source seq ${want.seq}); a retry re-plans`, { retryAfterS: 60 });
++ }
++ }
++ }
++ await timeline.replace(clipObjectId, timeline.SOURCE, rows, { jobId });
++ });
++ return { rows: rows.length };
++}
++
++/**
++ * Materialize `clip` over `vod`'s source timeline. → { ok: true, duration, end_ms } or null when the path does not
++ * apply (the flag off, no job context, a source without a ready object or media timeline, an empty window). Throws
++ * JobError on a failed encode/upload/pin. The bytes are placed before the pin; a throw before the commit removes
++ * what was staged and leaves the clip's rows as they were.
++ */
++async function materialize({ job, ctx, clip, vod }) {
++ // A clip that already has a file of its own never becomes a timeline clip: its recut stays a cut (F3.5).
++ if (!job || !ctx || !clip || !clip.object_id || clip.file_path || !vod || !vod.object_id) return null;
++ if (!config.hls.enabled || !config.hls.materialized) return null;
++ const srcObj = await model.getObject(vod.object_id);
++ if (!srcObj || srcObj.lifecycle_status !== 'ready') return null;
++ const srcRows = await timeline.list(vod.object_id, timeline.SOURCE);
++ const plan = planWindow(srcRows, Math.round((Number(clip.start_time) || 0) * 1000), Math.round((Number(clip.end_time) || 0) * 1000));
++ if (!plan) return null;
++ const clipObj = await model.getObject(clip.object_id);
++ if (!clipObj || clipObj.app_id !== clip.app_id) return null;
++ const vodStorage = require('./vod-storage');
++ const dir = derive.workDir(job.id);
++ const prev = new Map((await timeline.list(clip.object_id, timeline.SOURCE)).map((r) => [Number(r.seq), r]));
++ const target = await timeline.durableProvider(clipObj);
++ const remoteConfigured = vodStorage.REMOTE_PROVIDERS.some((p) => vodStorage.providerConfigured(p));
++ const stagedRemote = [];
++ const stagedLocal = [];
++ const failed = [];
++ let committed = false;
++ try {
++ const maxSeg = srcRows.reduce((m, r) => Math.max(m, Number(r.byte_length) || 0), 0);
++ segments.checkDisk(4 * maxSeg + 32 * 1024 * 1024);
++ const edgeSegments = new Map(); // clip seq -> placed media row fields
++ const edgeInits = new Map(); // init name -> placed init row fields
++ for (const [i, piece] of plan.pieces.entries()) {
++ if (piece.kind === 'copy') continue;
++ segments.aborted(ctx);
++ const seq = i + 1;
++ const made = await encodeEdge({ ctx, piece, initRow: plan.sourceInit, dir: path.join(dir, `edge-${piece.kind}-${seq}`) });
++ edgeSegments.set(seq, await placeFile({ clipObj, prev, seq, name: timeline.segmentName(seq), file: made.segFile,
++ contentType: 'video/iso.segment', target, remoteConfigured, stagedLocal, stagedRemote, failed, ctx }));
++ edgeInits.set(piece.init, await placeFile({ clipObj, prev, seq: piece.init === timeline.HEAD_INIT ? -1 : -2, name: piece.init, file: made.initFile,
++ contentType: 'video/mp4', target, remoteConfigured, stagedLocal, stagedRemote, failed, ctx }));
++ }
++ const rows = [];
++ for (const [i, piece] of plan.pieces.entries()) {
++ const seq = i + 1;
++ if (piece.kind === 'copy') { rows.push(copiedRow(piece.source, seq, plan.originMs)); continue; }
++ rows.push({ seq, name: timeline.segmentName(seq), start_ms: piece.start_ms - plan.originMs, end_ms: piece.end_ms - plan.originMs,
++ keyframe_ms: piece.start_ms - plan.originMs, ...edgeSegments.get(seq), init_name: piece.init });
++ }
++ if (edgeInits.has(timeline.HEAD_INIT)) rows.push({ seq: -1, name: timeline.HEAD_INIT, start_ms: 0, end_ms: 0, keyframe_ms: null, ...edgeInits.get(timeline.HEAD_INIT), init_name: null });
++ if (edgeInits.has(timeline.TAIL_INIT)) rows.push({ seq: -2, name: timeline.TAIL_INIT, start_ms: 0, end_ms: 0, keyframe_ms: null, ...edgeInits.get(timeline.TAIL_INIT), init_name: null });
++ if (plan.pieces.some((p) => p.kind === 'copy') && plan.sourceInit) rows.push(copiedInitRow(plan.sourceInit));
++ rows.sort((a, b) => Number(a.seq) - Number(b.seq));
++ segments.aborted(ctx);
++ await pinRows({ clipObjectId: clip.object_id, sourceObjectId: vod.object_id, rows, sourceRows: plan.sourceRows, jobId: job.id });
++ committed = true;
++ // Phase two, only once the rows are published: the edge bytes a previous attempt named that this plan does
++ // not. A location any row still names — the source's, or the source's interior locations the clip's own
++ // copied rows still name — is kept (F3.4), never deleted.
++ const named = new Set(rows.map((r) => `${r.key}\n${r.local_path || ''}`));
++ const stale = [...prev.values()].filter((old) => !named.has(`${old.key}\n${old.local_path || ''}`));
++ const { failed: stuck, kept } = await timeline.deleteBytes(stale);
++ if (stuck.length) console.warn(`[Materialize] ${job.id}: ${stuck.length} old edge segment(s) left behind (the storage orphan report names them), first ${stuck[0].provider}:${stuck[0].key}`);
++ if (kept.length) console.warn(`[Materialize] ${job.id}: ${kept.length} old edge location(s) kept — another object's rows name them`);
++ if (failed.length) throw new JobError('upload_failed', `${failed.length} edge segment(s) not durable yet (${failed[0].slice(0, 200)}); a retry uploads only those`, { retryAfterS: 600 });
++ return { ok: true, duration: plan.durationMs / 1000, end_ms: plan.endMs };
++ } catch (err) {
++ if (!committed) {
++ // Content-addressed: another run may have committed these very keys and files, so only unnamed ones go.
++ for (const k of stagedRemote) {
++ if (!await timeline.isNamed({ provider: k.provider, key: k.key }).catch(() => true)) await vodStorage.deleteObject(k.provider, k.key).catch(() => { /* best effort */ });
++ }
++ for (const p of stagedLocal) {
++ if (await timeline.isNamed({ localPath: p }).catch(() => true)) continue;
++ try { fs.unlinkSync(p); } catch { /* already gone */ }
++ }
++ }
++ throw err;
++ } finally {
++ derive.cleanupWork(job.id);
++ }
++}
++
++module.exports = { EDGE_MS, LOCATION_FIELDS, locationOf, planWindow, pinRows, materialize };
ddiff --git a/server/vod/clips-routes.js b/server/vod/clips-routes.js
iindex ebd7417..75801d5 100644
---- a/server/vod/clips-routes.js
++++ b/server/vod/clips-routes.js
@@@ -451,6 +451,18 @@ router.delete('/:id', tenantAuth({ verb: 'delete' }), async (req, res) => {
require('./vod-storage').deleteVodObjects(clip).catch(err =>
console.warn(`[Clips] Remote object cleanup failed for clip ${clip.id}:`, err.message));
}
++ // The clip's own bytes are its media_timeline rows (F3.5/F3.6): remove them and the bytes only it names. A
++ // location another object's rows still name — a materialized clip's interior, the source's segments — is kept
++ // (timeline.removeObject, F3.4), so deleting a clip never deletes its source's bytes. Every clip object is
++ // asked, not only a `timeline` one: a materialized clip that got a file of its own keeps its provider 'local'
++ // while its rows are still there (removeObject returns early when it has none, so a normal file clip is a no-op).
++ if (clip.object_id) {
++ const tl = await require('../objects/timeline').removeObject(clip.object_id)
++ .catch(err => { console.warn(`[Clips] Timeline cleanup failed for clip ${clip.id}:`, err.message); return null; });
++ // A durable delete that failed keeps its rows (removeObject) so it is not lost; the clips row (and its
++ // object) go below, and the hourly object sweep retries the rows for a projected object whose row is gone.
++ if (tl && tl.pending) console.warn(`[Clips] Clip ${clip.id}: ${tl.pending} timeline segment row(s) kept — their durable copy could not be deleted; the object sweep retries`);
++ }
await db.run('DELETE FROM clips WHERE id = ?', [clip.id]); // its object is marked deleted by the row-delete trigger, same statement
res.json({ message: 'Clip deleted' });
ddiff --git a/test/materialized-clip.test.js b/test/materialized-clip.test.js
nnew file mode 100644
iindex 0000000..502bb22
---- /dev/null
++++ b/test/materialized-clip.test.js
@@@ -0,0 +1,671 @@
++'use strict';
++// Materialized clips over the source's bytes (F3.6; docs/materialized-clips.md). A clip over a source that has a
++// source-rendition timeline, with MEDIA_HLS_ENABLED + MEDIA_MATERIALIZED_CLIPS, gets its own persisted media_timeline
++// rows: the interior segments copy the source's rows — packed chunk ranges included, no bytes written — and only the
++// two window edges are re-encoded from the source segment's bytes into fMP4 with their own inits (init-head.mp4 /
++// init-tail.mp4 on negative seq). The playlist switches init with #EXT-X-DISCONTINUITY + #EXT-X-MAP; the segment route
++// serves every init row as video/mp4; sprites decode each piece with its own init; readiness and the verify job treat
++// the clip as virtual; download?format=mp4 answers the playlist; deleting the clip removes its own rows/edge bytes and
++// keeps every location the source still names; a source removal racing the insert aborts the insert (or, committed
++// first, keeps the bytes the clip names); a source without a timeline, or the flag off, still gets the full re-encode.
++const assert = require('assert');
++const fs = require('fs');
++const os = require('os');
++const path = require('path');
++const http = require('http');
++const crypto = require('crypto');
++const { spawn, spawnSync } = require('child_process');
++
++(async () => {
++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-matclip-'));
++ const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
++ Object.assign(process.env, {
++ VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'), THUMBNAILS_PATH: dir('thumbnails'),
++ PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'), MEDIA_PUBLIC_URL: 'https://media.test',
++ MEDIA_JOBS_POLL_MS: '50', MEDIA_INVARIANT_SCAN_HOURS: '0', MEDIA_UPLOAD_MIN_FREE_MB: '0',
++ MEDIA_HLS_ENABLED: '1', MEDIA_MATERIALIZED_CLIPS: '1',
++ });
++ for (const p of ['B2', 'R2']) for (const k of ['ENDPOINT', 'BUCKET', 'KEY_ID', 'APP_KEY', 'ACCESS_KEY_ID', 'SECRET_ACCESS_KEY']) process.env[`MEDIA_${p}_${k}`] = '';
++
++ const sha = (buf) => crypto.createHash('sha256').update(buf).digest('hex');
++ const hasFfmpeg = spawnSync('ffmpeg', ['-version']).status === 0;
++ const hasX264 = hasFfmpeg && /libx264/.test(String(spawnSync('ffmpeg', ['-hide_banner', '-encoders']).stdout));
++
++ (async () => {
++ const config = require('../server/config');
++ assert.strictEqual(config.hls.enabled, true);
++ assert.strictEqual(config.hls.materialized, true, 'MEDIA_MATERIALIZED_CLIPS reads true');
++ const db = require('../server/db/database');
++ const timeline = require('../server/objects/timeline');
++ const model = require('../server/objects/model');
++ const mat = require('../server/vod/clip-materialize');
++ const previews = require('../server/jobs/previews');
++
++ // ── planWindow: edges over the source's media segments, the 100 ms boundary rule, the empty window ──
++ const hrow = (seq, start, end, extra = {}) => ({ seq, name: timeline.segmentName(seq), start_ms: start, end_ms: end, keyframe_ms: seq ? start : null,
++ key: `s/source/${timeline.segmentName(seq)}`, local_path: null, durable_provider: null, packed_object_id: null, byte_offset: null,
++ byte_length: 1000, sha256: 'a'.repeat(64), durability: 'local', init_name: null, ...extra });
++ const hand = [hrow(0, 0, 0), hrow(1, 0, 3000), hrow(2, 3000, 4500), hrow(3, 4500, 6000), hrow(4, 6000, 9000)];
++ const kinds = (p) => p.pieces.map((x) => x.kind);
++ {
++ const p = mat.planWindow(hand, 3101, 7899);
++ assert.deepStrictEqual(kinds(p), ['head', 'copy', 'tail'], 'a window cutting both ends re-encodes exactly the two edges');
++ assert.deepStrictEqual([p.pieces[0].start_ms, p.pieces[0].end_ms, p.pieces[0].init], [3101, 4500, timeline.HEAD_INIT]);
++ assert.strictEqual(Number(p.pieces[1].source.seq), 3, 'the interior piece is the source segment wholly inside the window, copied');
++ assert.deepStrictEqual([p.pieces[2].start_ms, p.pieces[2].end_ms, p.pieces[2].init], [6000, 7899, timeline.TAIL_INIT]);
++ assert.strictEqual(p.originMs, 3101);
++ assert.strictEqual(p.durationMs, 7899 - 3101, 'the source segment holding the end is cut at the request');
++ assert.deepStrictEqual(p.sourceRows.map((r) => Number(r.seq)), [0, 3], 'the pin names the seq 0 init and every copied segment (not the re-encoded edges)');
++ }
++ {
++ const p = mat.planWindow(hand, 3000, 5900);
++ assert.deepStrictEqual(kinds(p), ['copy', 'copy'], 'both edges within 100 ms of a boundary: nothing is re-encoded');
++ assert.strictEqual(p.originMs, 3000, 'the clip starts on the boundary');
++ assert.strictEqual(p.durationMs, 3000, 'and ends on the boundary (the snap is < 100 ms each side)');
++ }
++ assert.deepStrictEqual(kinds(mat.planWindow(hand, 3000, 5899)), ['copy', 'tail'], 'only the end cuts');
++ assert.deepStrictEqual(kinds(mat.planWindow(hand, 3101, 6000)), ['head', 'copy'], 'only the start cuts');
++ {
++ const p = mat.planWindow(hand, 3200, 4300);
++ assert.deepStrictEqual(kinds(p), ['head'], 'a window inside one segment is re-encoded once, as the head edge');
++ assert.deepStrictEqual([p.pieces[0].start_ms, p.pieces[0].end_ms], [3200, 4300]);
++ assert.deepStrictEqual(p.sourceRows, [], 'no copied row, so nothing to pin');
++ assert.strictEqual(mat.planWindow([], 0, 1000), null);
++ assert.strictEqual(mat.planWindow(hand, 3000, 3000), null, 'a zero-length window plans nothing');
++ assert.strictEqual(mat.planWindow(hand, 9000, 12000), null, 'a window past the timeline plans nothing');
++ }
++ {
++ // The exact rule is "more than 100 ms": 101 cuts, 100 snaps, and a window inside one segment is one piece.
++ const p = mat.planWindow(hand, 3101, 4399);
++ assert.deepStrictEqual([kinds(p), p.pieces[0].start_ms, p.pieces[0].end_ms, p.pieces[0].init],
++ [['head'], 3101, 4399, timeline.HEAD_INIT], 'both edges inside one segment: re-encoded once, as the head edge');
++ assert.deepStrictEqual(kinds(mat.planWindow(hand, 3100, 4400)), ['copy'], 'exactly 100 ms inside a boundary snaps to it');
++ assert.deepStrictEqual(kinds(mat.planWindow(hand, 3100, 6000)), ['copy', 'copy'], 'and the boundary itself never cuts');
++ }
++ console.log('✅ planWindow: head/interior/tail, the 100 ms rule, same-segment windows, empty windows');
++
++ // ── mediaPlaylist: byte-for-byte unchanged without edge inits; discontinuity + fresh map with them ──
++ {
++ const frozen = ['#EXTM3U', '#EXT-X-VERSION:7', '#EXT-X-TARGETDURATION:3', '#EXT-X-MEDIA-SEQUENCE:1',
++ '#EXT-X-PLAYLIST-TYPE:VOD', '#EXT-X-INDEPENDENT-SEGMENTS', '#EXT-X-MAP:URI="init.mp4"',
++ '#EXTINF:3.000,', '000001.m4s', '#EXTINF:1.500,', '000002.m4s', '#EXT-X-ENDLIST', ''].join('\n');
++ assert.strictEqual(timeline.mediaPlaylist([hrow(0, 0, 0), hrow(1, 0, 3000), hrow(2, 3000, 4500)]), frozen,
++ 'a playlist with no edge inits is byte for byte what it always was');
++ const edge = [
++ { ...hrow(-2, 0, 0), name: 'init-tail.mp4' }, { ...hrow(-1, 0, 0), name: 'init-head.mp4' }, hrow(0, 0, 0),
++ hrow(1, 0, 1000, { init_name: 'init-head.mp4' }), hrow(2, 1000, 3000), hrow(3, 3000, 4000, { init_name: 'init-tail.mp4' }),
++ ];
++ const pl = timeline.mediaPlaylist(edge);
++ const events = [];
++ for (const line of pl.split('\n')) {
++ if (line === '#EXT-X-DISCONTINUITY') events.push('disc');
++ const m = /^#EXT-X-MAP:URI="([^"]+)"/.exec(line);
++ if (m) events.push(`map ${m[1]}`);
++ }
++ assert.deepStrictEqual(events, ['map init-head.mp4', 'disc', 'map init.mp4', 'disc', 'map init-tail.mp4'], pl);
++ assert.strictEqual((pl.match(/#EXT-X-DISCONTINUITY/g) || []).length, 2, 'exactly two discontinuities');
++ assert.strictEqual((pl.match(/#EXTINF:/g) || []).length, 3, 'every segment is listed once');
++ }
++ {
++ // A rendition that produced an init and no media row still emits its EXT-X-MAP, byte for byte as before.
++ const frozenNoSeg = ['#EXTM3U', '#EXT-X-VERSION:7', '#EXT-X-TARGETDURATION:1', '#EXT-X-MEDIA-SEQUENCE:1',
++ '#EXT-X-PLAYLIST-TYPE:VOD', '#EXT-X-INDEPENDENT-SEGMENTS', '#EXT-X-MAP:URI="init.mp4"', '#EXT-X-ENDLIST', ''].join('\n');
++ assert.strictEqual(timeline.mediaPlaylist([hrow(0, 0, 0)]), frozenNoSeg,
++ 'an init with no segments still emits its EXT-X-MAP');
++ }
++ console.log('✅ mediaPlaylist: byte-for-byte without edge inits; two discontinuities with the right maps with them; an init-only rendition keeps its map');
++
++ // ── clipRows refuses a virtual clip over a materialized one (the simplest correct choice) ──
++ {
++ assert.deepStrictEqual(timeline.clipRows([
++ { ...hrow(-1, 0, 0), name: 'init-head.mp4' }, hrow(0, 0, 0), hrow(1, 0, 3000, { init_name: 'init-head.mp4' }),
++ ], 0, 3000), [], 'a source with edge inits is refused');
++ assert.deepStrictEqual(timeline.clipRows([hrow(0, 0, 0), hrow(1, 0, 3000)], 0, 3000).map((r) => Number(r.seq)), [0, 1],
++ 'a copy-only clip (edges on boundaries) still builds a virtual one');
++ }
++
++ // ── replace() drops negative-seq rows a new plan does not carry; the CHECK admits only the two edge init names ──
++ {
++ const edgeInit = { ...hrow(-1, 0, 0), name: 'init-head.mp4', key: 'c/source/init-head.mp4' };
++ assert.deepStrictEqual((await timeline.replace('mat-replace', 'source', [hrow(0, 0, 0), hrow(1, 0, 3000), edgeInit])).inserted, 3);
++ assert.deepStrictEqual((await timeline.list('mat-replace')).map((r) => Number(r.seq)), [-1, 0, 1]);
++ assert.strictEqual((await timeline.replace('mat-replace', 'source', [hrow(0, 0, 0), hrow(1, 0, 3000)])).removed, 1,
++ 'a replace that carries no edge init drops it');
++ assert.deepStrictEqual((await timeline.list('mat-replace')).map((r) => Number(r.seq)), [0, 1]);
++ assert.ok((await timeline.list('mat-replace')).every((r) => r.init_name == null), 'init_name defaults NULL');
++ await timeline.removeObject('mat-replace');
++ await assert.rejects(db.run(`INSERT INTO media_timeline (object_id, rendition, seq, name, start_ms, end_ms, key, byte_length, sha256)
++ VALUES ('mat-bad', 'source', -3, '000003.m4s', 0, 0, 'k', 1, 'x')`), /check|constraint/i, 'a negative seq outside the edge inits is rejected');
++ await assert.rejects(db.run(`INSERT INTO media_timeline (object_id, rendition, seq, name, start_ms, end_ms, key, byte_length, sha256)
++ VALUES ('mat-bad', 'source', -1, 'evil.mp4', 0, 0, 'k', 1, 'x')`), /check|constraint/i, 'a negative seq not named an edge init is rejected');
++ // Re-applying the migration is a no-op (as migration 0003's test does; only on PGlite, where the runner is the owner).
++ if (process.env.MEDIA_TEST_STORE !== 'pg') await db.getDb().query(fs.readFileSync(path.join(__dirname, '..', 'migrations', '0005_materialized_clips.sql'), 'utf8'));
++ }
++ console.log('✅ migration 0005 / replace: edge init rows admitted, dropped when unnamed, never beyond the two names');
++
++ // ── The sprite job picks each segment's init by init_name, falling back to seq 0 ──
++ {
++ const rows = [
++ { seq: -1, name: 'init-head.mp4' }, { seq: -2, name: 'init-tail.mp4' }, { seq: 0, name: 'init.mp4' },
++ { seq: 1, name: '000001.m4s', init_name: 'init-head.mp4' }, { seq: 2, name: '000002.m4s', init_name: null }, { seq: 3, name: '000003.m4s', init_name: 'init-tail.mp4' },
++ ];
++ assert.strictEqual(previews.initRowFor(rows, rows[3]).name, 'init-head.mp4', 'a head edge decodes with init-head');
++ assert.strictEqual(previews.initRowFor(rows, rows[4]).name, 'init.mp4', 'a NULL init_name falls back to seq 0');
++ assert.strictEqual(previews.initRowFor(rows, rows[5]).name, 'init-tail.mp4', 'a tail edge decodes with init-tail');
++ assert.strictEqual(previews.initRowFor([{ seq: 1, name: '000001.m4s', init_name: null }], { seq: 1, name: '000001.m4s', init_name: null }), null, 'no init row at all → null');
++ }
++
++ // ── The pin: a source removal or a moved location between plan and pin aborts; committed first, it keeps bytes ──
++ const seedSource = async (id) => {
++ const obj = { app_id: 'live', id };
++ const payloads = [Buffer.from('init-bytes'), Buffer.from('segment-one-bytes-'), Buffer.from('segment-two-bytes--'), Buffer.from('segment-three-bytes')];
++ const rows = [];
++ for (let seq = 0; seq < payloads.length; seq++) {
++ const bytes = payloads[seq];
++ const s = sha(bytes);
++ const name = timeline.segmentName(seq);
++ const key = timeline.keyFor(id, 'source', name, s.slice(0, 12));
++ const local = timeline.localPathFor(obj, 'source', name, s.slice(0, 12));
++ fs.mkdirSync(path.dirname(local), { recursive: true });
++ fs.writeFileSync(local, bytes);
++ const r = { seq, name, start_ms: seq ? (seq - 1) * 3000 : 0, end_ms: seq * 3000, keyframe_ms: seq ? (seq - 1) * 3000 : null,
++ key, local_path: local, durable_provider: null, packed_object_id: null, byte_offset: null, byte_length: bytes.length, sha256: s, durability: 'local', init_name: null };
++ await db.run(`INSERT INTO media_timeline (object_id, rendition, seq, ${timeline.FIELDS.join(', ')})
++ VALUES (?, ?, ?, ${timeline.FIELDS.map(() => '?').join(', ')})`, [id, 'source', seq, ...timeline.FIELDS.map((f) => r[f] ?? null)]);
++ rows.push(r);
++ }
++ return rows;
++ };
++ const copyRow = (source, seq, originMs) => {
++ const row = { seq };
++ for (const f of timeline.FIELDS) row[f] = source[f] == null ? null : source[f];
++ row.name = timeline.segmentName(seq);
++ row.start_ms = Number(source.start_ms) - originMs;
++ row.end_ms = Number(source.end_ms) - originMs;
++ row.keyframe_ms = source.keyframe_ms == null ? null : Number(source.keyframe_ms) - originMs;
++ row.init_name = null;
++ return row;
++ };
++ const planRows = (plan) => {
++ const out = [];
++ for (const [i, p] of plan.pieces.entries()) if (p.kind === 'copy') out.push(copyRow(p.source, i + 1, plan.originMs));
++ if (plan.pieces.some((p) => p.kind === 'copy') && plan.sourceInit) {
++ const r = copyRow(plan.sourceInit, 0, 0);
++ r.name = timeline.INIT_NAME; r.start_ms = 0; r.end_ms = 0; r.keyframe_ms = null;
++ out.push(r);
++ }
++ return out;
++ };
++ {
++ const srcA = await seedSource('mat-race-A');
++ const planA = mat.planWindow(srcA, 3000, 6000);
++ assert.deepStrictEqual(planA.pieces.map((p) => p.kind), ['copy'], 'the race window is interior-only');
++ const rowsA = planRows(planA);
++ await timeline.removeObject('mat-race-A');
++ await assert.rejects(mat.pinRows({ clipObjectId: 'mat-clip-A', sourceObjectId: 'mat-race-A', rows: rowsA, sourceRows: planA.sourceRows, jobId: 'mjob_RACEA' }),
++ (err) => err.code === 'media.timeline.changed' && err.retryAfterS > 0, 'a source removed between plan and pin aborts the insert, retryably');
++ assert.strictEqual((await timeline.list('mat-clip-A')).length, 0, 'nothing was inserted');
++
++ const srcB = await seedSource('mat-race-B');
++ const planB = mat.planWindow(srcB, 3000, 6000);
++ const rowsB = planRows(planB);
++ // A pack commits first: the planned row now names a chunk range (a moved location, same sha).
++ await db.run(`UPDATE media_timeline SET key = 'mat-race-B/source/chunk', packed_object_id = ?, byte_offset = 0 WHERE object_id = 'mat-race-B' AND seq = 2`, ['c'.repeat(64)]);
++ await assert.rejects(mat.pinRows({ clipObjectId: 'mat-clip-B', sourceObjectId: 'mat-race-B', rows: rowsB, sourceRows: planB.sourceRows, jobId: 'mjob_RACEB' }),
++ (err) => err.code === 'media.timeline.changed', 'a pack between plan and pin aborts the insert');
++ assert.strictEqual((await timeline.list('mat-clip-B')).length, 0, 'nothing was inserted');
++ await timeline.removeObject('mat-race-B');
++
++ // Committed first, then the source removed: the clip keeps its rows and the bytes it names.
++ const srcC = await seedSource('mat-race-C');
++ const planC = mat.planWindow(srcC, 0, 6000);
++ const rowsC = planRows(planC);
++ await mat.pinRows({ clipObjectId: 'mat-clip-C', sourceObjectId: 'mat-race-C', rows: rowsC, sourceRows: planC.sourceRows, jobId: 'mjob_RACEC' });
++ assert.deepStrictEqual((await timeline.list('mat-clip-C')).map((r) => Number(r.seq)), [0, 1, 2], 'the clip committed its rows');
++ const inWindow = srcC.filter((r) => Number(r.seq) <= 2).map((r) => r.local_path); // the init and the copied segments
++ const outside = srcC.filter((r) => Number(r.seq) === 3).map((r) => r.local_path); // not named by the clip
++ assert.ok(inWindow.concat(outside).every((f) => fs.existsSync(f)));
++ await timeline.removeObject('mat-race-C');
++ assert.strictEqual((await timeline.list('mat-race-C')).length, 0, 'the source rows are gone');
++ assert.ok(inWindow.every((f) => fs.existsSync(f)), 'the locations the clip still names are kept');
++ assert.ok(outside.every((f) => !fs.existsSync(f)), 'the locations no row names go with the source');
++ assert.ok(timeline.mediaPlaylist(await timeline.list('mat-clip-C')).includes('000002.m4s'), 'and the clip still writes its playlist from them');
++ await timeline.removeObject('mat-clip-C');
++ assert.ok(inWindow.every((f) => !fs.existsSync(f)), 'the last naming object deletes them');
++ }
++ console.log('✅ pin: a source removal or a pack between plan and pin aborts retryably; committed first, the bytes stay');
++
++ if (!hasFfmpeg || !hasX264) { console.log('materialized clips: pure checks passed; ffmpeg/libx264 sections skipped (not found)'); process.exit(0); }
++
++ // ── The ffmpeg path: a window over a packed source — two edges re-encoded, interior referenced ──
++ const blobStore = new Map();
++ const vodStorage = require('../server/vod/vod-storage');
++ Object.assign(vodStorage, {
++ providerConfigured: (p) => ['b2', 'r2'].includes(p),
++ providerAvailable: (p) => ['b2', 'r2'].includes(p),
++ uploadFile: async (p, key, file) => { blobStore.set(`${p}:${key}`, fs.readFileSync(file)); },
++ sha256Object: async (p, key) => (blobStore.has(`${p}:${key}`) ? { sha256: sha(blobStore.get(`${p}:${key}`)), size: blobStore.get(`${p}:${key}`).length } : null),
++ deleteObject: async (p, key) => { blobStore.delete(`${p}:${key}`); return true; },
++ presignGet: async () => null,
++ });
++ timeline.durableProvider = async () => 'b2'; // the placement router's choice, stubbed as the provider is
++
++ const stub = http.createServer((req, res) => {
++ req.resume();
++ req.on('end', () => {
++ res.setHeader('Content-Type', 'application/json');
++ if (req.url === '/oauth/token') return res.end(JSON.stringify({ access_token: 'tok', token_type: 'Bearer', expires_in: 300 }));
++ res.statusCode = 404; res.end('{}');
++ });
++ });
++ await new Promise((r) => stub.listen(0, '127.0.0.1', r));
++ const events = require('../server/events');
++ events.init({ eventsUrl: `http://127.0.0.1:${stub.address().port}`, clientSecret: 's', networkUrl: `http://127.0.0.1:${stub.address().port}`, intervalMs: 60000 });
++ await db.upsertApp({ app_id: 'live', api_key: 'live-key-mat' });
++ require('../server/thumbnails/thumbnail-service').generateClipThumbnail = async () => null;
++ const queue = require('../server/jobs/queue');
++ const worker = require('../server/jobs/worker');
++ const express = require('express');
++ const app = express();
++ app.use(express.json());
++ app.use('/api/v1/:app/vods', require('../server/vod/routes'));
++ app.use('/api/v1/:app/clips', require('../server/vod/clips-routes'));
++ app.use('/api/v2/:app/objects', require('../server/objects/routes'));
++ app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));
++ app.use('/o', require('../server/objects/routes').publicRouter);
++ app.use('/', require('../server/public/routes'));
++ const server = http.createServer(app);
++ await new Promise((r) => server.listen(0, '127.0.0.1', r));
++ const api = `http://127.0.0.1:${server.address().port}`;
++ const auth = { authorization: 'Bearer live-key-mat' };
++ const call = async (method, p, body) => {
++ const headers = { ...auth };
++ if (body && !(body instanceof FormData)) headers['content-type'] = 'application/json';
++ const res = await fetch(api + p, { method, headers, body: body && !(body instanceof FormData) ? JSON.stringify(body) : body, redirect: 'manual' });
++ return { status: res.status, body: await res.json().catch(() => null) };
++ };
++ const get = async (url, headers = {}) => {
++ const u = new URL(url, api);
++ const res = await fetch(api + u.pathname + u.search, { headers, redirect: 'manual' });
++ return { status: res.status, location: res.headers.get('location'), headers: res.headers, buf: Buffer.from(await res.arrayBuffer()) };
++ };
++ const signal = () => new AbortController().signal;
++ // Duplicate-window detection matches a recent clip within 8 s (start) / 10 s (end): age the rows first,
++ // as the other clip tests do, so each create below really creates.
++ const freshClip = async (body) => { await db.run("UPDATE clips SET created_at = '2000-01-01 00:00:00'"); return await call('POST', '/api/v1/live/clips', body); };
++ const runClip = async (jobId) => { assert.ok(await worker.runNow(jobId), 'the clip.cut job runs'); return queue.jobPublic(await queue.get(jobId)); };
++
++ // A 34 s, 160x90, 10 fps source with keyframes every 2 s: object.cmaf cuts ~2 s segments, object.pack one chunk.
++ const srcFile = path.join(process.env.VOD_PATH, 'mat-source.mp4');
++ const mk = spawnSync('nice', ['-n', '15', 'ffmpeg', '-v', 'error', '-y', '-f', 'lavfi', '-i', 'testsrc=duration=34:size=160x90:rate=10',
++ '-f', 'lavfi', '-i', 'sine=frequency=440:duration=34', '-c:v', 'libx264', '-preset', 'ultrafast', '-pix_fmt', 'yuv420p', '-g', '20', '-c:a', 'aac', '-shortest', srcFile]);
++ assert.strictEqual(mk.status, 0, String(mk.stderr));
++ const vodId = (await db.createVod({ app_id: 'live', user_id: 5, title: 'Mat source', file_path: srcFile,
++ file_size: fs.statSync(srcFile).size, duration_seconds: 34, visibility: 'public' })).lastInsertRowid;
++ const vod = await db.getVodById(vodId, 'live');
++ await db.run("UPDATE media_objects SET lifecycle_status = 'ready' WHERE id = ?", [vod.object_id]);
++ await model.upsertLocation(vod.object_id, { provider: 'local', key: srcFile, state: 'present', size_bytes: fs.statSync(srcFile).size, verified: true });
++ const cmafOut = await require('../server/jobs/cmaf').spec.run({ id: 'mjob_MATCMAF', app_id: 'live', object_id: vod.object_id, params: { segment_seconds: 2 } }, { signal: signal() });
++ assert.strictEqual(cmafOut.segments, 17, JSON.stringify(cmafOut));
++ const packOut = await require('../server/jobs/pack').spec.run({ id: 'mjob_MATPACK', app_id: 'live', object_id: vod.object_id, params: { target_seconds: 60 } }, { signal: signal() });
++ assert.strictEqual(packOut.packs, 1, JSON.stringify(packOut));
++ const srcRows = await timeline.list(vod.object_id, 'source');
++ const srcSegs = srcRows.filter((x) => Number(x.seq) > 0);
++ assert.ok(srcSegs.every((x) => x.packed_object_id && x.durability === 'durable'), 'every source segment is packed');
++ const chunkKey = srcSegs[0].key;
++ const chunkLocal = srcSegs[0].local_path;
++ assert.ok(blobStore.has(`b2:${chunkKey}`) && fs.existsSync(chunkLocal), 'the chunk is here and durable');
++ const segAt = (t) => srcSegs.find((x) => Number(x.start_ms) <= t && t < Number(x.end_ms));
++ const mid = (x) => Math.floor((Number(x.start_ms) + Number(x.end_ms)) / 2);
++ console.log(`✅ a 34 s source is cut into ${srcSegs.length} segments and packed into one chunk`);
++
++ // ── Materialize a ~31 s window: exactly two edges re-encoded, the interior rows name the chunk's bytes ──
++ const headSeg = srcSegs[0];
++ const tailSeg = segAt(Number(headSeg.start_ms) + 30000);
++ const startMs = mid(headSeg);
++ const endMs = mid(tailSeg);
++ const tailSeq = Number(tailSeg.seq);
++ assert.ok(tailSeq > 3 && endMs - startMs > 25000, 'the window spans most of the source');
++ const derive = require('../server/jobs/derive');
++ const realFfmpeg = derive.ffmpeg;
++ let encodes = 0;
++ derive.ffmpeg = async (...a) => { encodes++; return await realFfmpeg(...a); };
++ let r = await freshClip({ vod_id: vodId, start_s: startMs / 1000, end_s: endMs / 1000, materialize: true, title: 'Materialized window' });
++ assert.strictEqual(r.status, 202, JSON.stringify(r.body));
++ const clipId = r.body.id;
++ const job = await runClip(r.body.job_id);
++ assert.strictEqual(job.status, 'succeeded', JSON.stringify(job.error));
++ assert.strictEqual(encodes, 2, 'exactly two segments re-encoded (the two edges)');
++ const clip = await db.getClipById(clipId);
++ const clipObj = clip.object_id;
++ assert.deepStrictEqual([clip.status, clip.file_path, clip.storage_provider], ['ready', null, 'timeline']);
++ assert.strictEqual(Number(clip.duration_seconds), (endMs - startMs) / 1000, 'the clip duration is the planned window');
++ const md = model.parseJson((await model.getObject(clipObj)).metadata, {});
++ assert.deepStrictEqual([md.virtual, md.materialized], [true, true], 'virtual stays, materialized is added');
++
++ const expectedSeqs = [-2, -1, 0, ...Array.from({ length: tailSeq }, (_, i) => i + 1)];
++ const rows = await timeline.list(clipObj, 'source');
++ assert.deepStrictEqual(rows.map((x) => Number(x.seq)), expectedSeqs, 'the edge inits, the source init and the pieces, all numbered');
++ const bySeq = new Map(rows.map((x) => [Number(x.seq), x]));
++ const srcBySeq = new Map(srcRows.map((x) => [Number(x.seq), x]));
++ for (let s = 2; s < tailSeq; s++) {
++ const c = bySeq.get(s);
++ const o = srcBySeq.get(s);
++ assert.deepStrictEqual([c.key, c.local_path, c.durable_provider, c.packed_object_id, Number(c.byte_offset), Number(c.byte_length), c.sha256, c.init_name],
++ [o.key, o.local_path, o.durable_provider, o.packed_object_id, Number(o.byte_offset), Number(o.byte_length), o.sha256, null],
++ `interior seq ${s} names the source's chunk bytes`);
++ assert.deepStrictEqual([Number(c.start_ms), Number(c.end_ms)], [Number(o.start_ms) - startMs, Number(o.end_ms) - startMs], 'times are relative to the window');
++ }
++ assert.deepStrictEqual([bySeq.get(1).init_name, bySeq.get(tailSeq).init_name], [timeline.HEAD_INIT, timeline.TAIL_INIT], 'the two edges name their own init');
++ assert.deepStrictEqual([Number(bySeq.get(1).start_ms), Number(bySeq.get(1).end_ms)], [0, Number(headSeg.end_ms) - startMs], 'the head edge ends at its source segment\'s end');
++ assert.deepStrictEqual([Number(bySeq.get(tailSeq).start_ms), Number(bySeq.get(tailSeq).end_ms)], [Number(tailSeg.start_ms) - startMs, endMs - startMs], 'the tail edge starts at its source segment\'s start');
++ for (const s of [1, tailSeq]) {
++ const e = bySeq.get(s);
++ assert.ok(e.key.startsWith(`${clipObj}/source/`) && e.local_path.startsWith(path.join(path.resolve(process.env.OBJECTS_PATH), '.timeline', 'live', clipObj, 'source')), `edge ${s} is the clip's own`);
++ assert.strictEqual(sha(fs.readFileSync(e.local_path)), e.sha256, `edge ${s} sha256`);
++ assert.ok(blobStore.has(`b2:${e.key}`), `edge ${s} is durable too`);
++ }
++ for (const initSeq of [-1, -2]) {
++ const init = bySeq.get(initSeq);
++ assert.strictEqual(sha(fs.readFileSync(init.local_path)), init.sha256);
++ assert.ok(blobStore.has(`b2:${init.key}`), `${init.name} is durable`);
++ }
++ console.log('✅ materialize: the interior rows copy the packed source; exactly two edge segments and two edge inits are its own');
++
++ // ── The playlist: two discontinuities with the right maps; every row serves through /o/<clip>/source/<name> ──
++ const pl = (await get(`/o/${clipObj}/source/index.m3u8`)).buf.toString();
++ {
++ const events = [];
++ for (const line of pl.split('\n')) {
++ if (line === '#EXT-X-DISCONTINUITY') events.push('disc');
++ const m = /^#EXT-X-MAP:URI="([^"]+)"/.exec(line);
++ if (m) events.push(`map ${m[1]}`);
++ }
++ assert.deepStrictEqual(events, ['map init-head.mp4', 'disc', 'map init.mp4', 'disc', 'map init-tail.mp4'], pl);
++ assert.strictEqual((pl.match(/#EXTINF:/g) || []).length, tailSeq, 'every piece is listed once');
++ assert.ok(pl.includes('#EXT-X-MEDIA-SEQUENCE:1') && pl.trim().endsWith('#EXT-X-ENDLIST'));
++ }
++ for (const row of rows) {
++ const s = await get(`/o/${clipObj}/source/${row.name}`);
++ assert.strictEqual(s.status, 200, row.name);
++ assert.strictEqual(sha(s.buf), row.sha256, `${row.name} serves its indexed bytes`);
++ assert.strictEqual(s.headers.get('content-type'), Number(row.seq) <= 0 ? 'video/mp4' : 'video/iso.segment', `${row.name} MIME`);
++ }
++ const probed = await new Promise((resolve) => {
++ const p = spawn('nice', ['-n', '15', 'ffprobe', '-v', 'quiet', '-rw_timeout', '10000000', '-show_entries', 'format=duration', '-of', 'csv=p=0', `${api}/o/${clipObj}/source/index.m3u8`]);
++ let out = '';
++ p.stdout.on('data', (d) => { out += d; });
++ p.on('close', () => resolve(out.trim()));
++ });
++ assert.ok(Math.abs(Number(probed) - (endMs - startMs) / 1000) < 0.2, `ffprobe plays across both discontinuities: ${JSON.stringify(probed)}`);
++ console.log('✅ the playlist carries two discontinuities with the right maps; every segment and edge init serves with the right MIME');
++
++ // ── Readiness counts it playable; the verify job skips it ──
++ const readiness = require('../server/objects/readiness');
++ const rdy = await readiness.compute(await model.getObject(clipObj));
++ assert.deepStrictEqual([rdy.playable, rdy.bytes_verified, rdy.reason], [true, true, null], 'a materialized clip is playable with no location of its own');
++ assert.ok(await db.get(`SELECT 1 FROM clips WHERE id = ? AND ${readiness.playableSql('clips.object_id')}`, [clipId]), 'playableSql agrees');
++ const verified = await require('../server/objects/verify-job').runOnce({ batch: 100, head: async () => undefined, upload: async () => { throw new Error('no upload in this test'); }, maxReuploads: 0 });
++ assert.ok(!verified.objects.some((o) => o.object_id === clipObj), 'the verify job skips a materialized clip as it skips a virtual one');
++ r = await call('PUT', `/api/v1/live/clips/${clipId}`, { title: 'Materialized, renamed' });
++ assert.strictEqual(r.status, 200, JSON.stringify(r.body));
++ const remd = model.parseJson((await model.getObject(clipObj)).metadata, {});
++ assert.deepStrictEqual([remd.virtual, remd.materialized], [true, true], 'a re-projection keeps virtual and materialized');
++
++ // ── An edge-only clip (a window inside one source segment) decodes through the sprite with its own init ──
++ {
++ const soloSeg = srcSegs[4];
++ const sStart = Number(soloSeg.start_ms) + 300;
++ const sEnd = Number(soloSeg.end_ms) - 1;
++ r = await freshClip({ vod_id: vodId, start_s: sStart / 1000, end_s: sEnd / 1000, materialize: true, title: 'Edge only' });
++ assert.strictEqual(r.status, 202, JSON.stringify(r.body));
++ assert.strictEqual((await runClip(r.body.job_id)).status, 'succeeded');
++ const soloObj = (await db.getClipById(r.body.id)).object_id;
++ const soloRows = await timeline.list(soloObj, 'source');
++ assert.deepStrictEqual(soloRows.map((x) => Number(x.seq)), [-1, 1], 'one head edge, its own init, no seq 0 (nothing copied)');
++ assert.strictEqual(soloRows.find((x) => Number(x.seq) === 1).init_name, timeline.HEAD_INIT);
++ const soloPl = (await get(`/o/${soloObj}/source/index.m3u8`)).buf.toString();
++ assert.ok(soloPl.includes('#EXT-X-MAP:URI="init-head.mp4"') && !soloPl.includes('#EXT-X-DISCONTINUITY'), soloPl);
++ // Run the job directly: the worker's ctx.saveCheckpoint runs the callback (no job checkpoint needed here).
++ const spriteCtx = { signal: signal(), saveCheckpoint: async (_cp, alsoInTx) => { if (alsoInTx) await alsoInTx(); } };
++ const sprite = await previews.sprite.run({ id: 'mjob_MATSPRITE', app_id: 'live', object_id: soloObj, params: { frames: 4, columns: 2, tile_width: 96 } }, spriteCtx);
++ assert.deepStrictEqual([sprite.sprite.count, sprite.sprite.columns, sprite.sprite.rows], [1, 1, 1]);
++ const spriteFile = (await model.listLocations(sprite.object_id))[0].key;
++ const rr = spawnSync('ffmpeg', ['-v', 'error', '-i', spriteFile, '-frames:v', '1', '-vf', 'scale=1:1,format=gray', '-f', 'rawvideo', '-'], { encoding: null, maxBuffer: 1 << 20 });
++ const grey = rr.stdout && rr.stdout.length ? rr.stdout[rr.stdout.length - 1] : -1;
++ assert.ok(grey > 30, `the edge decodes with init-head through the sprite (grey ${grey})`);
++ }
++ console.log('✅ an edge-only clip plays and its sprite decodes the edge with init-head');
++
++ // ── Downloads: format=mp4 answers the playlist (no remux it cannot read); /c/:id still 302s to it ──
++ r = await call('GET', `/api/v2/live/objects/${clipObj}/download?format=mp4`);
++ assert.strictEqual(r.status, 200, JSON.stringify(r.body));
++ assert.strictEqual(r.body.url, `${config.publicUrl}/o/${clipObj}/master.m3u8`);
++ assert.strictEqual(r.body.public, true);
++ assert.strictEqual((await db.all("SELECT id FROM media_jobs WHERE job_type = 'object.remux'")).length, 0, 'no remux is queued for a timeline-only clip');
++ const redir = await fetch(`${api}/api/v2/live/objects/${clipObj}/download?format=mp4&redirect=1`, { headers: auth, redirect: 'manual' });
++ assert.strictEqual(redir.status, 302);
++ assert.ok(String(redir.headers.get('location')).endsWith(`/o/${clipObj}/master.m3u8`), redir.headers.get('location'));
++ const c = await get(`/c/${clipId}`);
++ assert.deepStrictEqual([c.status, new URL(c.location).pathname], [302, `/o/${clipObj}/master.m3u8`], '/c/:id redirects a file-less materialized clip to its playlist');
++ console.log('✅ download?format=mp4 and /c/:id answer the playlist for a materialized clip');
++
++ // ── A virtual clip is still the default; recutting it with materialize:true materializes it ──
++ r = await freshClip({ vod_id: vodId, start_s: 3000 / 1000, end_s: 6000 / 1000, title: 'Virtual then materialized' });
++ assert.strictEqual(r.status, 201, JSON.stringify(r.body));
++ assert.deepStrictEqual([r.body.status, r.body.storage_provider, r.body.file_path], ['ready', 'timeline', null]);
++ const virtId = r.body.id;
++ const virtObj = (await db.getClipById(virtId)).object_id;
++ assert.strictEqual((await timeline.list(virtObj, 'source')).length, 0, 'a virtual clip has no rows of its own');
++ r = await call('POST', `/api/v1/live/clips/${virtId}/recut`, { materialize: true });
++ assert.strictEqual(r.status, 202, JSON.stringify(r.body));
++ assert.strictEqual((await runClip(r.body.job_id)).status, 'succeeded');
++ assert.ok((await timeline.list(virtObj, 'source')).some((x) => Number(x.seq) < 0), 'materializing a virtual clip writes its own edge init rows');
++ assert.strictEqual(model.parseJson((await model.getObject(virtObj)).metadata, {}).materialized, true);
++ assert.strictEqual((await call('DELETE', `/api/v1/live/clips/${virtId}`)).status, 200, 'and it deletes like any materialized clip');
++ assert.strictEqual((await timeline.list(virtObj, 'source')).length, 0);
++ console.log('✅ a virtual clip is the default; recut?materialize=1 materializes it over the source');
++
++ // ── The flag off, and a source without a timeline: the full re-encode, exactly as before ──
++ const cutter = require('../server/vod/clip-cutter');
++ const realCut = cutter.cutClipFile;
++ const cuts = [];
++ cutter.cutClipFile = async (o) => {
++ cuts.push(o);
++ const f = path.join(process.env.CLIPS_PATH, `fallback-${cuts.length}.webm`);
++ fs.writeFileSync(f, Buffer.alloc(2048, 7));
++ return { ok: true, filePath: f, duration: 4 };
++ };
++ config.hls.materialized = false;
++ r = await freshClip({ vod_id: vodId, start_s: 20, end_s: 24, materialize: true, title: 'Flag off' });
++ assert.strictEqual(r.status, 202, JSON.stringify(r.body));
++ assert.strictEqual((await runClip(r.body.job_id)).status, 'succeeded');
++ assert.strictEqual(cuts.length, 1, 'MEDIA_MATERIALIZED_CLIPS off: clip.cut re-encodes as before');
++ const offClip = await db.getClipById(r.body.id);
++ assert.ok(offClip.file_path && fs.existsSync(offClip.file_path), 'and the clip has its own file');
++ assert.strictEqual((await timeline.list(offClip.object_id, 'source')).length, 0, 'no timeline rows');
++
++ // With the flag back on, a clip that has a file of its own keeps being cut: never turned into a timeline clip.
++ config.hls.materialized = true;
++ r = await call('POST', `/api/v1/live/clips/${offClip.id}/recut`);
++ assert.strictEqual(r.status, 202, JSON.stringify(r.body));
++ assert.strictEqual((await runClip(r.body.job_id)).status, 'succeeded');
++ assert.strictEqual(cuts.length, 2, 'a clip with a file is re-cut, not materialized');
++ const recut = await db.getClipById(offClip.id);
++ assert.ok(recut.file_path && fs.existsSync(recut.file_path), 'it still has a file of its own');
++ assert.strictEqual((await timeline.list(recut.object_id, 'source')).length, 0, 'and no timeline rows');
++
++ const plainFile = path.join(process.env.VOD_PATH, 'plain.mp4');
++ fs.copyFileSync(srcFile, plainFile);
++ const plainVodId = (await db.createVod({ app_id: 'live', user_id: 5, title: 'No timeline', file_path: plainFile,
++ file_size: fs.statSync(plainFile).size, duration_seconds: 34, visibility: 'public' })).lastInsertRowid;
++ const plainVod = await db.getVodById(plainVodId, 'live');
++ await db.run("UPDATE media_objects SET lifecycle_status = 'ready' WHERE id = ?", [plainVod.object_id]);
++ await model.upsertLocation(plainVod.object_id, { provider: 'local', key: plainFile, state: 'present', size_bytes: fs.statSync(plainFile).size, verified: true });
++ r = await freshClip({ vod_id: plainVodId, start_s: 1, end_s: 5, materialize: true, title: 'No timeline' });
++ assert.strictEqual(r.status, 202, JSON.stringify(r.body));
++ assert.strictEqual((await runClip(r.body.job_id)).status, 'succeeded');
++ assert.strictEqual(cuts.length, 3, 'a source without a timeline re-encodes in full');
++ cutter.cutClipFile = realCut;
++ console.log('✅ flag off, or a source without a timeline: the full re-encode stays exactly as it was');
++
++ // ── Review follow-ups: a materialized clip re-cut to a file (1), its renditions (2), its end_time (3),
++ // the pending durable delete on delete (4) and the pin's lock order (5) ──
++ {
++ // Finding 3: a window whose head snaps back to its segment boundary (start within EDGE_MS after it) and
++ // whose tail cuts. plan.originMs < startTime, so the old startTime + duration overshot the snapped end.
++ const seg3 = srcSegs[1];
++ const boundary3 = Number(seg3.start_ms);
++ const start3 = boundary3 + 50;
++ const end3 = mid(srcSegs[4]);
++ const plan3 = mat.planWindow(srcRows, start3, end3);
++ assert.ok(plan3 && plan3.originMs === boundary3 && plan3.endMs === end3,
++ `the head snaps back to ${boundary3} and the tail cuts at ${end3}: ${plan3 && JSON.stringify([plan3.originMs, plan3.endMs])}`);
++ r = await freshClip({ vod_id: vodId, start_s: start3 / 1000, end_s: end3 / 1000, materialize: true, title: 'Snapped start' });
++ assert.strictEqual(r.status, 202, JSON.stringify(r.body));
++ const snapId = r.body.id;
++ assert.strictEqual((await runClip(r.body.job_id)).status, 'succeeded');
++ const snap = await db.getClipById(snapId);
++ const snapObj = snap.object_id;
++ assert.strictEqual(Number(snap.end_time), plan3.endMs / 1000, 'finding 3: end_time is the snapped end');
++ assert.notStrictEqual(Number(snap.end_time), start3 / 1000 + plan3.durationMs / 1000, 'finding 3: not the drifted sum');
++ assert.strictEqual(Number(snap.duration_seconds), plan3.durationMs / 1000, 'the duration is the planned window');
++ console.log('✅ finding 3: end_time equals the snapped end, not startTime + duration');
++
++ const ownRows = (await timeline.list(snapObj, 'source')).filter((x) => String(x.key || '').startsWith(`${snapObj}/source/`));
++ assert.ok(ownRows.length >= 2, 'the clip has its own edge rows');
++ assert.ok(ownRows.every((x) => fs.existsSync(x.local_path) && blobStore.has(`b2:${x.key}`)), 'the edge bytes exist before the re-cut');
++
++ // Finding 2: with renditions on, a materialized clip queues none — its job could not read a file it has not got.
++ const tq = require('../server/objects/timeline-queue');
++ const hls = require('../server/objects/hls');
++ const wasRenditions = config.hls.renditions;
++ config.hls.renditions = true;
++ assert.strictEqual(config.hls.renditions, true, 'renditions really are on');
++ assert.strictEqual(tq.isVirtualClip(await model.getObject(snapObj)), true, 'the clip object is recognised as playing without a file');
++ assert.strictEqual(await tq.queueRendition('live', snapObj), null, 'finding 2: queueRendition queues nothing');
++ await hls.discoveryForObject(await model.getObject(snapObj));
++ assert.strictEqual((await db.all("SELECT id FROM media_jobs WHERE job_type = 'rendition.create' AND object_id = ?", [snapObj])).length, 0,
++ 'finding 2: the lazy HLS path queues nothing either');
++ config.hls.renditions = wasRenditions;
++ console.log('✅ finding 2: a materialized clip queues no rendition');
++
++ // Finding 1: re-cut to a file drops the timeline rows, the edge bytes and the materialized marker; the
++ // stale playlist stops being served and /c/:id answers the file.
++ const wasMaterialized = config.hls.materialized;
++ config.hls.materialized = false;
++ const realCut2 = cutter.cutClipFile;
++ const cutFile2 = path.join(process.env.CLIPS_PATH, 'snap-recut.webm');
++ cutter.cutClipFile = async () => { fs.writeFileSync(cutFile2, Buffer.alloc(4096, 9)); return { ok: true, filePath: cutFile2, duration: plan3.durationMs / 1000 }; };
++ r = await call('POST', `/api/v1/live/clips/${snapId}/recut`, { materialize: true });
++ assert.strictEqual(r.status, 202, JSON.stringify(r.body));
++ assert.strictEqual((await runClip(r.body.job_id)).status, 'succeeded');
++ cutter.cutClipFile = realCut2;
++ config.hls.materialized = wasMaterialized;
++ const recut1 = await db.getClipById(snapId);
++ assert.deepStrictEqual([recut1.file_path, recut1.storage_provider], [cutFile2, 'local'], 'the clip is now a file clip');
++ assert.strictEqual((await timeline.list(snapObj, 'source')).length, 0, 'finding 1: no timeline rows remain');
++ const md1 = model.parseJson((await model.getObject(snapObj)).metadata, {});
++ assert.strictEqual(md1.materialized, undefined, 'finding 1: metadata.materialized is gone');
++ assert.strictEqual(md1.virtual, undefined, 'and it is no longer marked virtual');
++ assert.ok(ownRows.every((x) => !fs.existsSync(x.local_path) && !blobStore.has(`b2:${x.key}`)), 'finding 1: the edge bytes are deleted');
++ assert.strictEqual((await get(`/o/${snapObj}/source/init-head.mp4`)).status, 404, 'finding 1: HLS no longer serves the stale rows');
++ const played = await get(`/c/${snapId}`);
++ assert.strictEqual(played.status, 200, 'finding 1: /c/:id serves the file');
++ assert.ok(!String(played.location || '').includes('master.m3u8'), 'finding 1: and not the stale playlist');
++ console.log('✅ finding 1: re-cutting a materialized clip to a file releases its rows, edge bytes and marker');
++
++ // Finding 1 (DELETE): the route asks removeObject whenever the clip has an object_id, not only when its
++ // provider is 'timeline' (a materialized clip re-cut to a file keeps provider 'local' with rows left).
++ const dd = path.join(path.resolve(process.env.OBJECTS_PATH), '.timeline', 'live', snapObj, 'source');
++ fs.mkdirSync(dd, { recursive: true });
++ const leftover = path.join(dd, '000001.m4s');
++ fs.writeFileSync(leftover, Buffer.from('leftover'));
++ await db.run(`INSERT INTO media_timeline (object_id, rendition, seq, ${timeline.FIELDS.join(', ')})
++ VALUES (?, 'source', 1, ${timeline.FIELDS.map(() => '?').join(', ')})`,
++ [snapObj, '000001.m4s', 0, 1000, 0, 'snap/leftover', leftover, null, null, null, 8, sha(Buffer.from('leftover')), 'local', null]);
++ assert.strictEqual((await timeline.list(snapObj, 'source')).length, 1, 'a row is planted for a non-timeline clip');
++ assert.strictEqual((await call('DELETE', `/api/v1/live/clips/${snapId}`)).status, 200);
++ assert.strictEqual((await timeline.list(snapObj, 'source')).length, 0, 'finding 1: DELETE removes the rows of a clip whose provider is not timeline');
++ assert.ok(!fs.existsSync(leftover), 'finding 1: and its bytes');
++ console.log('✅ finding 1: DELETE calls removeObject for any clip with an object_id');
++
++ // Finding 4: a durable delete that fails on clip delete is logged, and the object sweep retries it.
++ r = await freshClip({ vod_id: vodId, start_s: 12, end_s: 16, title: 'Pending delete' });
++ assert.strictEqual(r.status, 201, JSON.stringify(r.body));
++ const pendId = r.body.id;
++ const pendObj = (await db.getClipById(pendId)).object_id;
++ const pendKey = 'f4/pending-segment';
++ const pendBytes = Buffer.from('pending-durable-bytes');
++ blobStore.set(`b2:${pendKey}`, pendBytes);
++ await db.run(`INSERT INTO media_timeline (object_id, rendition, seq, ${timeline.FIELDS.join(', ')})
++ VALUES (?, 'source', 1, ${timeline.FIELDS.map(() => '?').join(', ')})`,
++ [pendObj, '000001.m4s', 0, 1000, 0, pendKey, null, 'b2', null, null, pendBytes.length, sha(pendBytes), 'durable', null]);
++ const realDelete = vodStorage.deleteObject;
++ vodStorage.deleteObject = async (p, k) => (p === 'b2' && k === pendKey ? false : realDelete(p, k));
++ const logged = [];
++ const warn = console.warn;
++ console.warn = (...a) => { logged.push(a.join(' ')); };
++ let del;
++ try { del = await call('DELETE', `/api/v1/live/clips/${pendId}`); } finally { console.warn = warn; vodStorage.deleteObject = realDelete; }
++ assert.strictEqual(del.status, 200, JSON.stringify(del.body));
++ assert.ok(logged.some((w) => /timeline segment row\(s\) kept/.test(w)), `finding 4: the pending delete is logged: ${logged.join(' | ')}`);
++ assert.strictEqual((await timeline.list(pendObj, 'source')).length, 1, 'finding 4: the row survives with its key');
++ assert.ok(!(await db.get('SELECT 1 AS x FROM clips WHERE id = ?', [pendId])), 'the clip row is gone');
++ const swept = await timeline.sweepPendingRemovals();
++ assert.ok(swept.objects >= 1 && swept.removed >= 1, `finding 4: the object sweep retries it: ${JSON.stringify(swept)}`);
++ assert.strictEqual((await timeline.list(pendObj, 'source')).length, 0, 'finding 4: the retry drops the row');
++ assert.ok(!blobStore.has(`b2:${pendKey}`), 'finding 4: and its durable bytes');
++ console.log('✅ finding 4: a pending durable delete is logged and the object sweep retries it');
++
++ // Finding 5: the pin's locking read takes the source rows in seq order (the order markPacked and a source
++ // DELETE lock them), so it cannot deadlock with them.
++ const matSrc = fs.readFileSync(path.join(__dirname, '..', 'server', 'vod', 'clip-materialize.js'), 'utf8');
++ assert.ok(/seq IN \([\s\S]*\) ORDER BY seq FOR SHARE/.test(matSrc), 'finding 5: the locking select is ORDER BY seq');
++ console.log('✅ finding 5: the pin locks the source rows ORDER BY seq');
++ }
++
++ // ── Deletion: the clip's rows and edge bytes go; the source's bytes stay; and vice versa ──
++ const edgeAssets = [bySeq.get(-1), bySeq.get(-2), bySeq.get(1), bySeq.get(tailSeq)];
++ const edgeGone = () => edgeAssets.every((x) => !fs.existsSync(x.local_path) && !blobStore.has(`b2:${x.key}`));
++ assert.ok(edgeAssets.every((x) => fs.existsSync(x.local_path) && blobStore.has(`b2:${x.key}`)), 'the clip\'s edge bytes exist before the delete');
++ const srcRowCount = (await timeline.list(vod.object_id, 'source')).length;
++ r = await call('DELETE', `/api/v1/live/clips/${clipId}`);
++ assert.strictEqual(r.status, 200, JSON.stringify(r.body));
++ assert.strictEqual((await timeline.list(clipObj, 'source')).length, 0, 'the clip\'s rows are gone');
++ assert.strictEqual((await timeline.list(vod.object_id, 'source')).length, srcRowCount, 'the source\'s rows stay');
++ assert.ok(fs.existsSync(chunkLocal) && blobStore.has(`b2:${chunkKey}`), 'the source\'s chunk stays');
++ assert.ok(edgeGone(), 'the clip\'s own edge bytes go');
++
++ const s2 = mid(srcSegs[1]);
++ const e2 = mid(srcSegs[3]);
++ r = await freshClip({ vod_id: vodId, start_s: s2 / 1000, end_s: e2 / 1000, materialize: true, title: 'Source removed' });
++ assert.strictEqual(r.status, 202, JSON.stringify(r.body));
++ assert.strictEqual((await runClip(r.body.job_id)).status, 'succeeded');
++ const clipBId = r.body.id;
++ const clipBObj = (await db.getClipById(clipBId)).object_id;
++ const hold = await model.placeHold({ object_id: vod.object_id, kind: 'admin', reason: 'materialized clip test' });
++ r = await call('DELETE', `/api/v1/live/clips/${clipBId}`);
++ assert.strictEqual(r.status, 409, JSON.stringify(r.body));
++ assert.ok((await timeline.list(clipBObj, 'source')).length > 0, 'a held source still freezes its clip');
++ await model.releaseHold(hold.id);
++ assert.ok((await timeline.removeObject(vod.object_id)).removed > 0, 'the source rows go');
++ const bRows = await timeline.list(clipBObj, 'source');
++ assert.ok(bRows.some((x) => Number(x.seq) < 0), 'the clip keeps its own rows');
++ assert.ok(fs.existsSync(chunkLocal) && blobStore.has(`b2:${chunkKey}`), 'and the chunk it names is kept');
++ assert.strictEqual((await get(`/o/${clipBObj}/source/index.m3u8`)).status, 200, 'the clip stays playable after the source is gone');
++ const bSeg = bRows.find((x) => Number(x.seq) === 2);
++ const served = await get(`/o/${clipBObj}/source/${bSeg.name}`);
++ assert.deepStrictEqual([served.status, sha(served.buf)], [200, bSeg.sha256], 'an interior segment still serves from the kept chunk');
++ r = await call('DELETE', `/api/v1/live/clips/${clipBId}`);
++ assert.strictEqual(r.status, 200, JSON.stringify(r.body));
++ assert.ok(!fs.existsSync(chunkLocal) && !blobStore.has(`b2:${chunkKey}`), 'the last naming object deletes the chunk, here and durable');
++ console.log('✅ deletion: a materialized clip releases only its own bytes; a removed source keeps the clip playable');
++
++ events._reset();
++ server.close();
++ stub.close();
++ console.log('materialized clips: all checks passed');
++ process.exit(0);
++ })().catch((err) => { console.error(err); process.exit(1); });
++})().catch((err) => { console.error(err); process.exit(1); });