openvibers/openvibe.media Commits
Wave 4: media job system, multipart and presigned uploads, R2 eviction drill
Changes
@@@ -68,3 +68,21 @@ MEDIA_R2_SECRET_ACCESS_KEY=# OpenVibe.Events (durable events): vod/clip/object/storage outcomes are also published through# Media's outbox with the OAuth client credentials (grant events.event.publish). Unset = off.#EVENTS_URL=http://127.0.0.1:4300++++# ── Object uploads (docs/object-model.md#presigned-uploads, #multipart-uploads) ──++# MEDIA_SIGNING_SECRET signs presigned upload/download URLs; set it in production.++#MEDIA_SIGNING_SECRET=++#MEDIA_OBJECT_MAX_MB=256 # single-part limit; larger objects use multipart++#MEDIA_UPLOAD_TOKEN_TTL_S=3600 # default presigned upload URL lifetime++#MEDIA_MULTIPART_MAX_MB=20480++#MEDIA_MULTIPART_PART_MB=64 # bounds: MEDIA_MULTIPART_MIN_PART_MB=5, MEDIA_MULTIPART_MAX_PART_MB=256++#MEDIA_MULTIPART_TTL_HOURS=24++#MEDIA_UPLOAD_MIN_FREE_MB=10240 # free disk kept in reserve by multipart uploads and split/remux jobs++++# ── Jobs (docs/object-model.md#jobs) ─────────────────────────++#MEDIA_JOBS_ENABLED=1++#MEDIA_JOBS_POLL_MS=5000++#MEDIA_JOBS_LIGHT_CONCURRENCY=2++#MEDIA_JOBS_HEAVY_CONCURRENCY=1++#MEDIA_JOBS_HEAVY_WHILE_RECORDING=0++#MEDIA_INVARIANT_SCAN_HOURS=24 # the size-invariant validator (proposes split/remux; never runs them)@@@ -41,11 +41,18 @@ server/thumbnails/ thumbnail service + /api/v1/:app/thumbnailspublic/routes.js public /v /c /p /t /fobjects/ canonical object model: model (projections, holds), routes (/api/v2 + /o),-- backfill, reconcile, invariant, signing, verify-job (scheduled copy-- verification), copy-report — see docs/object-model.md++ backfill, reconcile, invariant, signing (presigned URLs), multipart,++ content-type, verify-job (scheduled copy verification), copy-report —++ see docs/object-model.md++ jobs/ media_jobs: queue (states, idempotency, media.job.* events in the state++ change's transaction), worker (light/heavy lanes, leases, retries,++ cancellation), routes (/api/v2/:app/jobs), types: thumbnail.regenerate,++ invariant.scan (proposes split/remux), object.split, object.remux(openvibe-shared: pinned OpenVibe.Shared v1.0.0 release, installed by npm)scripts/smoke-test.sh end-to-end smoke test (boots a temp instance)scripts/backfill-objects.js / reconcile-objects.js / object-invariant.js / no-good-copy-report.js object-model operator tools++scripts/media-jobs.js list jobs, run the size-invariant scan (dry run by default), approve/cancel proposals++scripts/r2-eviction-drill.js evict one VOD's R2 copy, prove B2 serves it, re-warm R2; JSON artifact (dry run by default)```## Visitor sign-in@@@ -7,7 +7,10 @@ projection over an object through its `object_id` column. Every existing URL andunchanged. New code talks to objects through the [v2 API](#object-api-v2).Code: `server/objects/` (`model.js`, `routes.js`, `backfill.js`, `reconcile.js`, `invariant.js`,--`signing.js`, `verify-job.js`, `copy-report.js`). Schema: the bottom of `server/db/schema.sql`. Tests: `test/objects-*.test.js`.++`signing.js`, `verify-job.js`, `copy-report.js`, `multipart.js`, `content-type.js`) and the job system in++`server/jobs/` (`queue.js`, `worker.js`, `routes.js`, `types.js`, `thumbnail.js`, `invariant-scan.js`,++`derive.js`). Schema: the bottom of `server/db/schema.sql`. Tests: `test/objects-*.test.js`, `test/jobs*.test.js`,++`test/r2-eviction-drill.test.js`.## Tables@@@ -17,7 +20,8 @@ Code: `server/objects/` (`model.js`, `routes.js`, `backfill.js`, `reconcile.js`,| `media_locations` | one row per provider copy (`UNIQUE(object_id, provider)`): `provider` (`local` `b2` `r2`), `bucket`, `key` (absolute path for local, object key for B2/R2), `storage_class` (`hot` local, `cold` B2, `cache` R2), `state`, `checksum`, `size_bytes`, `verified_at` || `media_relationships` | `from_object_id`, `relation`, `to_object_id`, `metadata`. In use: `clip_of` (clip to vod, with start/end), `thumbnail_of` (thumbnail to vod/clip). Reserved: `derived_from`, `screenshot_of` || `media_variants` | `object_id`, `variant_name`, `derived_object_id`, `recipe`. In use: `thumbnail` |--| `media_jobs` | generic derivative/maintenance jobs (`job_type`, `status`, `attempts`, `checkpoint`, `error`). **Schema only in this pass:** nothing enqueues or consumes jobs yet |++| `media_jobs` | derivative/maintenance jobs: `id` (mjob_…), `app_id`, `object_id`, `job_type`, `status`, `idempotency_key`, `params`, `result`, `attempts`/`max_attempts`, `run_after`, `lease_until`, `checkpoint`, `error`/`error_code`, `cancel_requested`, `created_by`, `decided_by`. See [Jobs](#jobs) |++| `media_uploads`, `media_upload_parts` | multipart upload sessions and the parts received (size, sha256). See [Multipart uploads](#multipart-uploads) || `media_holds` | retention holds, see [Holds](#retention-holds) || `media_invariant_violations` | public playback objects over the size policy, see [Invariant](#public-object-size-invariant) |@@@ -129,9 +133,15 @@ All routes live under `/api/v2/:app/objects`.| method | path | notes ||---|---|---|--| POST | `/` | init. `{ kind, visibility (default private), mime_type, size_bytes, filename, content_hash (expected sha256), metadata (≤ 16 KB), user_id }` returns 201 `{ id, object, upload: { method: 'PUT', url, token, expires_at, max_bytes, complete_url } }`. Checks at init: `size_bytes` against `MEDIA_OBJECT_MAX_MB` (413), the quota including reservations (413 `media.quota.exceeded`), and the public-size invariant (422) |++| POST | `/` | init. `{ kind, visibility (default private), mime_type, size_bytes, filename, content_hash (expected sha256), metadata (≤ 16 KB), user_id, upload_ttl (60-86400 s), multipart, part_size }` returns 201 `{ id, object, upload: { method: 'PUT', url, token, expires_at, max_bytes, content_type, complete_url, multipart_url } }`. `url` is a [presigned PUT URL](#presigned-uploads). With `multipart: true` (required above `MEDIA_OBJECT_MAX_MB`): `method: 'multipart'`, `url: null`, and `upload.multipart` is the new [session](#multipart-uploads). Checks at init: `size_bytes` against `MEDIA_OBJECT_MAX_MB` (single part) or `MEDIA_MULTIPART_MAX_MB` (413), the quota including reservations (413 `media.quota.exceeded`), the public-size invariant (422) and the [content type](#content-types) for the kind (415 `media.object.unsupported_type`) |++| POST | `/:id/upload-url` | a fresh presigned single-PUT URL for an uploading object (`{ ttl }` 60-86400 s) || PUT | `/:id/content` | the bytes, single part. Auth is the upload token (`?token=` from init, or `X-Upload-Token`) or the usual credential. Streams to disk computing sha256. Refuses more than the declared size or the limit (413), a size mismatch (400) and quota overrun (413). Stored at `OBJECTS_PATH/<app>/<id>` with a verified local location. Mounted ahead of the JSON body parser, so any Content-Type is taken as raw bytes; the Content-Type becomes `mime_type` when init set none. Can be repeated while the object is `uploading` |--| POST | `/:id/complete` | checks the expected hash (422 `media.object.hash_mismatch`), the invariant and the quota. Sets `ready` and sends the `media.object.uploaded` webhook. Also accepts the upload token |++| POST | `/:id/complete` | checks the expected hash (422 `media.object.hash_mismatch`), the invariant, the quota and the bytes against the type (415 `media.object.content_mismatch`). Sets `ready` and sends the `media.object.uploaded` webhook. Also accepts the upload token |++| POST | `/:id/multipart` | start a [multipart session](#multipart-uploads) `{ part_size, size_bytes (when init declared none) }` → 201 |++| GET | `/:id/multipart/:uploadId` | the session: `parts` received (number, size, sha256), `missing`, `received_bytes`, `expires_at` |++| PUT | `/:id/multipart/:uploadId/parts/:n` | one part, raw bytes of exactly its size; optional `X-Content-SHA256` |++| POST | `/:id/multipart/:uploadId/complete` | `{ content_hash, parts: [{ part_number, sha256 }] }` (both optional): assembles, then the checks of `/complete` |++| DELETE | `/:id/multipart/:uploadId` | abort: parts deleted, the object stays `uploading` || GET | `/:id` | metadata: providers and states of each copy, never paths or keys || GET | `/` | cursor list, newest first. `?limit (≤200)&cursor&kind&visibility&status&owner (usr_…)&user_id&include_deleted` returns `{ objects, next_cursor }` || DELETE | `/:id` | soft delete: `lifecycle_status = deleted`, and the bytes are kept for `MEDIA_DELETE_RETENTION_DAYS` (default 30). 409 `media.object.held` under a hold. 409 `media.object.legacy_managed` for projected objects, which are deleted through their v1 route |@@@ -149,10 +159,71 @@ All routes live under `/api/v2/:app/objects`.- A local copy streams with Range support. Otherwise the route redirects 302 to a presigned R2 URL, then a presigned B2 URL.- Headers: `X-Content-Type-Options: nosniff` and `X-Robots-Tag: noindex`. Content is `inline` only for images (not SVG), video, audio, PDF and plain text, and `attachment` for everything else.--**Signing.** HMAC-SHA256 over the purpose (`get` or `put`), the object id and the expiry, keyed with--`MEDIA_SIGNING_SECRET`. A download signature cannot be used as an upload token, or the other way--round. Without the secret, a random per-process secret is used and a warning is logged, so links--stop working at the next restart. **Set `MEDIA_SIGNING_SECRET` in production.**++**Signing.** HMAC-SHA256 keyed with `MEDIA_SIGNING_SECRET` over a purpose and what the link may do:++downloads (`get`) over the object id and expiry; upload tokens (`put2`) over the tenant, the object,++its size and the expiry; multipart sessions (`mpart`) over the tenant, the object, the session, its++total size and the expiry. A signature for one purpose is never valid for another. Without the++secret, a random per-process secret is used and a warning is logged, so links stop working at the++next restart. **Set `MEDIA_SIGNING_SECRET` in production.**++++### Presigned uploads++++The `url` from init (or `POST /:id/upload-url`) is a presigned single-PUT URL:++`…/objects/<id>/content?token=v2.<exp>.<size>.<mac>`. No other credential is needed, so a browser++can send the bytes straight to Media. The token is scoped:++++- to the **tenant** and the **object**: a token for another object or another tenant never verifies,++ and the URL must address the object's tenant;++- to the **size** the object declared (the PUT must be exactly that many bytes). An object created++ without a size gets a size-0 token: any size up to `MEDIA_OBJECT_MAX_MB`;++- in **time**: `upload_ttl` / `ttl` (60 s to 24 h, default `MEDIA_UPLOAD_TOKEN_TTL_S`).++++The size is part of the signature, so it cannot be edited. Tokens of the earlier form `<exp>.<mac>`++(object only) are accepted until they expire. These are Media-signed URLs: the bytes land on Media's++disk, not directly in a bucket.++++### Multipart uploads++++For objects above `MEDIA_OBJECT_MAX_MB` (up to `MEDIA_MULTIPART_MAX_MB`, default 20 GB), or whenever a++client asks with `multipart: true`. Code: `server/objects/multipart.js`.++++1. **Initiate** at init (`multipart: true`) or with `POST /:id/multipart`. The object must declare its++ size. `part_size` is `MEDIA_MULTIPART_MIN_PART_MB` (5) to `MEDIA_MULTIPART_MAX_PART_MB` (256) MB,++ default `MEDIA_MULTIPART_PART_MB` (64), and grows if there would be more than 10,000 parts. The++ answer carries `upload_id`, `parts_expected`, and a session `token` (`mp1.…`) already in++ `part_url_template`, `status_url`, `complete_url` and `abort_url`. The quota is checked (the++ declared size is reserved from init), and there must be free disk for twice the size plus++ `MEDIA_UPLOAD_MIN_FREE_MB` (507 `media.storage.insufficient`). A new session replaces an open one.++2. **Upload parts** (`PUT …/parts/<n>`, any order, in parallel). Every part but the last is exactly++ `part_size` bytes and the last is the rest; any other size is refused (400/413). A part is written++ to a temp file and renamed when complete, so a dropped connection leaves nothing half-written.++ Sending a part again replaces it. `X-Content-SHA256` is checked when sent.++3. **Resume** after a drop: `GET` the session and send the parts in `missing`.++4. **Complete**: every part must be there (409 `media.upload.parts_missing` lists `missing`); a++ `parts` list with sha256 values must match what was received. The parts are concatenated into++ `OBJECTS_PATH/<app>/<id>` while hashing, the parts are deleted, and the checks of `/complete`++ follow (expected hash, invariant, quota, content). If one of those fails, the object stays++ `uploading` with its content and can be sent again.++5. **Abort** (`DELETE`) deletes the parts; the object stays `uploading`. A single-part PUT is refused++ while a session is open (409 `media.upload.multipart_active`).++++Sessions expire after `MEDIA_MULTIPART_TTL_HOURS` (24): the hourly purge deletes their parts and any++part directory no session owns. `media_uploads_open` on `/metrics` counts open sessions, and++reconciliation reports sessions left open past their expiry (`incomplete_multipart`).++++### Content types++++Code: `server/objects/content-type.js`.++++- **The type must suit the kind** (415 `media.object.unsupported_type`, at init or at the PUT when++ init declared none): `vod` video/* or audio/*, `clip` video/*, `thumbnail` / `screenshot` / `avatar`++ image/* but not SVG. `file` and `asset` take any well-formed type.++- **Inline-served bytes must be what their type says** (415 `media.object.content_mismatch` at++ complete): `/o/:id` serves image/*, video/*, audio/* and PDF inline, so for those types (and every++ kind with a rule) the first bytes must match a known signature of that family: JPEG, PNG, GIF, WebP,++ BMP, AVIF/HEIC; Matroska/WebM, MP4/MOV, Ogg, AVI, MPEG-TS; MP3, WAV, FLAC, M4A; PDF.++- Everything else is served as an attachment with `nosniff`, so its declared type is taken as given.++ The declared type wins over the PUT's `Content-Type` (declared at init: the PUT header is ignored).**Quota.** `apps.quota_bytes` is compared against v1 file bytes plus native objects that are`uploading` (their declared size counts as a reservation) or `ready`. Soft-deleted objects do not@@@ -204,9 +275,10 @@ The exit code is 1 when issues were found.| `orphan_location` | a location row whose object does not exist || `deleted_publicly_reachable` | a deleted object whose inherited row still serves it, or a thumbnail of a deleted object that is still served by name || `missing_projection` | an inherited row with no object yet (run the backfill) |++| `incomplete_multipart` | a multipart session still open past its expiry (the hourly purge should have removed it) |**Not covered yet:** orphan *provider* objects (bucket keys with no row, which needs bucket--listing) and incomplete multipart uploads. `server/objects/reconcile.js` takes a `head` function,++listing). `server/objects/reconcile.js` takes a `head` function,so tests run it against a fake provider.## Scheduled verification@@@ -260,9 +332,124 @@ A *public playback object* is a `ready` vod or clip whose visibility is `public`Lists every public playback object above target, largest first, records the rows, and resolves rows that no longer apply. `--dry-run` writes nothing.--**Nothing is re-encoded in this pass.** Most whole-stream VODs are far above 500 MB. They will show--up as violations until the segment-native model lands (roadmap W4 deliverable 7, capture in W7),--which splits playback into objects that fit.++- **Validator job.** `invariant.scan` (see [Jobs](#jobs)) runs per tenant every++ `MEDIA_INVARIANT_SCAN_HOURS` (24) and on demand. It records the violations and **proposes** one job++ per public playback object above the max: `object.split` into stream-copy parts of about the target++ size when the duration is known, `object.remux` when it is not (the remux writes the duration, so a++ split can be planned next). **Proposals never run by themselves**: they wait in status `proposed`++ until the owner approves or cancels them. A cancelled proposal is not proposed again; a proposal++ whose object stopped violating (made private, deleted) is withdrawn. `scripts/media-jobs.js scan`++ shows what would be proposed without writing anything.++++**Nothing is re-encoded, and nothing is made private automatically.** A split makes private parts next++to the unchanged source; the source stays a violation until its owner decides (make the parts public++and the source private, or leave it). The segment-native model (roadmap W4 deliverable 7, capture in++W7) is what will make every new recording fit.++++## Jobs++++Code: `server/jobs/`. Tests: `test/jobs.test.js`, `test/jobs-invariant.test.js`.++++**States.** `proposed` (waiting for the owner; the worker never takes it) → `queued` → `running` →++`succeeded` | `failed` | `cancelled`. A failed attempt goes back to `queued` with `run_after` (backoff++30 s, 2 min, 8 min, …, at most 1 h) until `max_attempts`; a handler can mark a failure permanent.++++**Events.** Every state change and its `media.job.<transition>` event (`proposed`, `queued`,++`started`, `retrying`, `succeeded`, `failed`, `cancelled`; subject `job <id>`, payload the job)++commit in one SQLite transaction through Media's outbox, the same rule as the outcome events++(`webhooks.announce()`): no change without its event, no event for a change that rolled back.++Progress events are `low` priority; outcomes and proposals are `important`. Job events go to++OpenVibe.Events only, not to app webhooks.++++**Idempotency.** `Idempotency-Key` (or `idempotency_key`) is unique per tenant: the same key with the++same request (type, object, params) answers with the job it made (`Idempotent-Replayed: true`); a++different request under a used key is 409 `media.job.idempotency_conflict`. At most 50 jobs per tenant++may be queued or running (429 `media.job.too_many`).++++**Cancellation, by the owner.** A proposed or queued job is cancelled at once. A running job gets++`cancel_requested`; the worker aborts it (its ffmpeg is killed) and it ends `cancelled`. Finished++jobs answer 409. The owner is the tenant (app key, service token for the namespace, or the project's++app token); a caller acting for one of the app's users (`X-OV-User-Id`) sees and decides only the jobs++it created and the jobs on objects it owns.++++**Worker.** In-process, polling every `MEDIA_JOBS_POLL_MS`, in two lanes: `light`++(`MEDIA_JOBS_LIGHT_CONCURRENCY`, 2) and `heavy` (`MEDIA_JOBS_HEAVY_CONCURRENCY`, 1; waits while a++recording runs unless `MEDIA_JOBS_HEAVY_WHILE_RECORDING=1`). A running job holds a lease++(`MEDIA_JOBS_LEASE_S`) renewed by a heartbeat; at start, jobs the previous process left `running` are++requeued (or failed when out of attempts). Handlers checkpoint progress and resume from it.++`MEDIA_JOBS_ENABLED=0` stops the worker. Finished thumbnail jobs are pruned after++`MEDIA_JOBS_RETENTION_DAYS`; other jobs are kept.++++| type | lane | what it does |++|---|---|---|++| `thumbnail.regenerate` | light | (Re)generates a vod/clip thumbnail from its media (local file or presigned cloud copy). `params { kind, id }`, or an object that is a projected vod/clip. Result `{ url }` |++| `invariant.scan` | light | The [size-invariant validator](#public-object-size-invariant): records violations, proposes `object.split` / `object.remux`, withdraws moot proposals. Tenant-wide (no object) |++| `object.split` | heavy | Stream-copies a vod/clip (or a video/audio object) into parts (`params.parts` 2-1000 or `segment_seconds`), each a new **private** object with a `derived_from` relationship (`job_id`, `part`, `start_seconds`, `duration_seconds`). Cuts land on keyframes, so neighbouring parts can overlap slightly. The source is never changed. Checkpointed per part |++| `object.remux` | heavy | Stream-copy remux of the whole source (seek index, duration, MP4 faststart) into one new private object; also the source's `remux` variant |++++Split and remux refuse a source that is not ready, a tenant quota the output would exceed, and too++little free disk (the source size plus `MEDIA_UPLOAD_MIN_FREE_MB`; retried after 30 min).++++**The v1 thumbnail route runs as a job.** `POST /api/v1/:app/thumbnails/:kind/:id` without an image++queues `thumbnail.regenerate` (joining one already queued or running for the same item, so concurrent++requests share one ffmpeg run), runs it at once and answers `{ url, job_id }` as before; 404 when++there is no media, 500 when no frame came out. `?async=1` answers 202 `{ job }` without waiting.++++**API** (`/api/v2/:app/jobs`, same auth as the objects API; writes need `media.object.upload`, reads++`media.object.read`):++++| method | path | notes |++|---|---|---|++| GET | `/` | cursor list, newest first: `?status&type&object_id&limit (≤200)&cursor` → `{ jobs, next_cursor }` |++| POST | `/` | `{ type, object_id, params, max_attempts }` + `Idempotency-Key` → 202 `{ job }` (200 on a replay) |++| GET | `/:jobId` | `{ job }` |++| POST | `/:jobId/approve` | a proposal → `queued` (409 `media.job.not_proposed` otherwise) |++| POST | `/:jobId/cancel`, DELETE `/:jobId` | 200 cancelled, 202 while a running job stops, 409 when finished |++++**Operator script.**++++```++node scripts/media-jobs.js list [--status proposed] [--type object.split] [--app live] [--json]++node scripts/media-jobs.js show <job id>++node scripts/media-jobs.js scan [--app live] [--apply] [--json] # dry run unless --apply++node scripts/media-jobs.js approve <job id>... [--by <who>]++node scripts/media-jobs.js cancel <job id>... [--by <who>] [--reason <text>]++```++++It changes the database directly (the service's worker picks approved jobs up on its next poll) and,++when the service's outbox is on, queues each change's event in the same transaction. It never runs a++job. `media_jobs{type,status}` on `/metrics` counts jobs.++++## R2 eviction drill++++The exit criterion "the canonical B2 object survives R2 eviction", as a repeatable drill with a kept++artifact. Code: `scripts/r2-eviction-drill.js`; test: `test/r2-eviction-drill.test.js` (a fake S3 and++a local Media, through the real storage engine).++++```++node scripts/r2-eviction-drill.js (--vod <id> | --pick) [--execute] [--base-url https://openvibe.media]++ [--no-http] [--max-mb 512] [--out artifact.json] [--json]++```++++- **Dry run by default:** it checks the preconditions and both copies, and writes the artifact. Nothing moves.++- **Preconditions** (else it refuses, exit 2): the VOD is served from R2 (`storage_provider = r2`), is++ not recording, is not held (a hold freezes placement), and is public or unlisted for the HTTP++ checks; B2 and R2 are configured; the B2 and R2 copies have the same size (and the recorded size)++ and the same first MiB. Evicting the cache in front of a corrupt canonical copy would lose the only++ good one. `--pick` takes the smallest eligible VOD up to `--max-mb`.++- **`--execute`**, through the storage engine's own tier moves: (1) `GET <base>/v/<id>?raw=1`++ redirects to R2 and serves the expected first MiB; (2) `demoteFromR2` deletes the R2 copy (after++ checking B2) and flips the row to `b2`; (3) R2 answers 404, and `/v/<id>` redirects to B2 serving the++ same first MiB; (4) `promoteToR2` copies B2 back to R2 (verified by HEAD) and flips the row to `r2`;++ (5) `/v/<id>` redirects to R2 again with the same bytes, and `media_locations` shows B2 and R2 present.++- A failed step stops the drill (exit 1). After the eviction the VOD stays served from B2, which the++ drill has just proven works. `--no-http` checks the playback decision in-process instead of++ `GET /v`, which otherwise counts as one view from the drill's host.++- **Artifact:** JSON (default `data/drills/r2-eviction-<vod>-<time>.json`) with every step, the HEAD++ sizes, the first-MiB sha256 of each copy and of what was served, `media_locations` before, after the++ eviction and after, and the verdict (`pass`, `fail`, `refused`, `dry-run`).++++Not run against production yet: it needs the owner's go-ahead.## Configuration@@@ -273,7 +460,18 @@ which splits playback into objects that fit.| `MEDIA_DELETE_RETENTION_DAYS` | 30 | soft-deleted native bytes are kept this long || `MEDIA_SIGNING_SECRET` | *(random per process)* | HMAC key for signed URLs and upload tokens || `MEDIA_SIGNED_URL_TTL_S` | 300 | default signed download lifetime |--| `MEDIA_UPLOAD_TOKEN_TTL_S` | 3600 | upload token lifetime |++| `MEDIA_UPLOAD_TOKEN_TTL_S` | 3600 | default presigned upload URL lifetime (60 s to 24 h per request) |++| `MEDIA_MULTIPART_MAX_MB` | 20480 | largest object a multipart upload may declare |++| `MEDIA_MULTIPART_MIN_PART_MB` / `_MAX_PART_MB` / `_PART_MB` | 5 / 256 / 64 | part size bounds and default |++| `MEDIA_MULTIPART_TTL_HOURS` | 24 | an unfinished multipart session is purged after this |++| `MEDIA_UPLOAD_MIN_FREE_MB` | 10240 | free disk kept in reserve by multipart uploads and split/remux jobs |++| `MEDIA_JOBS_ENABLED` | on | `0` stops the job worker |++| `MEDIA_JOBS_POLL_MS` | 5000 | worker poll interval |++| `MEDIA_JOBS_LIGHT_CONCURRENCY` / `_HEAVY_CONCURRENCY` | 2 / 1 | jobs per lane |++| `MEDIA_JOBS_HEAVY_WHILE_RECORDING` | off | `1` lets split/remux run while a recording is being written |++| `MEDIA_JOBS_LEASE_S` | 120 | running-job lease (renewed by the heartbeat) |++| `MEDIA_INVARIANT_SCAN_HOURS` | 24 | the size-invariant validator's schedule per tenant (`0` = on demand only) |++| `MEDIA_JOBS_RETENTION_DAYS` | 30 | finished thumbnail jobs are pruned after this || `MEDIA_PUBLIC_OBJECT_{MAX,TARGET,WARN}_MB` | 500 / 256 / 384 | invariant thresholds || `MEDIA_VERIFY_ENABLED` | on | `0` turns the scheduled verification off || `MEDIA_VERIFY_INTERVAL_MIN` | 10 | minutes between verification runs |@@@ -284,10 +482,11 @@ which splits playback into objects that fit.## Not in this pass--- Multipart uploads and presigned direct-to-bucket uploads. Native bytes are stored locally.++- Presigned direct-to-bucket uploads. Presigned URLs are Media-signed and native bytes are stored locally.- Tiering of native objects to B2/R2.- Copy/move aliases, lifecycle rules and an S3-compatible façade.--- A `media_jobs` worker.--- Orphan-bucket-object and incomplete-multipart detection.++- Clip cutting and the finalize remux still run inline, not as jobs (thumbnail regeneration was the first++ operation moved onto the job system).++- Orphan-bucket-object detection.- The segment-native timeline.- App assets (emotes and sounds, the `assets` table) are not projected yet. The `asset` kind exists for v2 uploads.nnew file mode 100644@@@ -0,0 +1,85 @@++#!/usr/bin/env node++'use strict';++/**++ * Media jobs, for the operator (docs/object-model.md#jobs). Works on the service's database directly;++ * the running service's worker picks up approved jobs on its next poll. When the service's Events++ * outbox is on (the event_outbox table exists), every state change made here queues its++ * media.job.* event there too, in the same transaction, and the service relays it.++ *++ * node scripts/media-jobs.js list [--status proposed] [--type object.split] [--app live] [--limit 50] [--json]++ * node scripts/media-jobs.js show <job id> [--json]++ * node scripts/media-jobs.js scan [--app live] [--apply] [--json]++ * the size-invariant validator; a dry run unless --apply (which records violations and proposals)++ * node scripts/media-jobs.js approve <job id>... [--by <who>] proposed -> queued++ * node scripts/media-jobs.js cancel <job id>... [--by <who>] [--reason <text>]++ * (every command takes --db ./data/media.db)++ *++ * Nothing here runs a job. Exit 0 on success, 1 when a named job could not be decided, 2 on usage errors.++ */++const path = require('path');++++const args = process.argv.slice(2);++const has = (n) => args.includes(`--${n}`);++const arg = (n, d) => { const i = args.indexOf(`--${n}`); return i >= 0 ? args[i + 1] : d; };++const VALUE_FLAGS = ['status', 'type', 'app', 'limit', 'by', 'reason', 'db'];++const positional = args.filter((a, i) => !a.startsWith('--') && !(i > 0 && VALUE_FLAGS.includes(String(args[i - 1]).replace(/^--/, ''))));++if (arg('db')) process.env.DB_PATH = path.resolve(arg('db'));++++const db = require('../server/db/database');++const events = require('../server/events');++const queue = require('../server/jobs/queue');++const scan = require('../server/jobs/invariant-scan');++++function usage(msg) {++ if (msg) console.error(msg);++ console.error('usage: media-jobs.js list|show|scan|approve|cancel … (see the header of this file)');++ process.exit(2);++}++++function line(j) {++ const size = j.params && j.params.source_size_bytes ? ` ${(j.params.source_size_bytes / 1048576).toFixed(0)} MB` : '';++ return `${j.id} ${j.status.padEnd(9)} ${j.type.padEnd(20)} ${j.app_id.padEnd(8)} ${j.object_id || '-'}${size}${j.error ? ` (${j.error})` : ''}`;++}++++(async () => {++ const [cmd, ...ids] = positional;++ const by = arg('by', `operator:${require('os').userInfo().username}`);++ events.initWriter();++ let exit = 0;++ if (cmd === 'list') {++ const apps = arg('app') ? [arg('app')] : db.all('SELECT DISTINCT app_id FROM media_jobs ORDER BY app_id').map(r => r.app_id);++ const out = [];++ for (const a of apps) out.push(...queue.list(a, { status: arg('status'), type: arg('type'), limit: arg('limit', 50) }).jobs.map(queue.jobPublic));++ console.log(has('json') ? JSON.stringify({ jobs: out, counts: queue.counts() }, null, 2) : [...out.map(line), `counts: ${JSON.stringify(queue.counts())}`].join('\n'));++ } else if (cmd === 'show') {++ const j = queue.jobPublic(queue.get(ids[0]));++ if (!j) usage(`no job ${ids[0]}`);++ console.log(JSON.stringify(j, null, 2));++ } else if (cmd === 'scan') {++ const apps = arg('app') ? [arg('app')] : db.all("SELECT DISTINCT app_id FROM media_objects WHERE kind IN ('vod', 'clip') ORDER BY app_id").map(r => r.app_id);++ const reports = {};++ for (const a of apps) reports[a] = scan.scanTenant(a, { dryRun: !has('apply') });++ if (has('json')) console.log(JSON.stringify(reports, null, 2));++ else {++ for (const [a, r] of Object.entries(reports)) {++ console.log(`${a}: ${r.violations} public object(s) over ${(r.thresholds.maxBytes / 1048576).toFixed(0)} MB; ${r.dry_run ? 'would propose' : 'proposed'} ${r.proposed}, already proposed ${r.already_proposed}, ${r.dry_run ? 'would withdraw' : 'withdrawn'} ${r.withdrawn} ${JSON.stringify(r.by_type)}`);++ }++ if (!has('apply')) console.log('(dry run: nothing recorded; --apply records the violations and the proposals)');++ }++ } else if (cmd === 'approve' || cmd === 'cancel') {++ if (!ids.length) usage(`${cmd}: name at least one job`);++ for (const id of ids) {++ const j = queue.get(id);++ if (!j) { console.error(`${id}: no such job`); exit = 1; continue; }++ if (cmd === 'approve') {++ const out = queue.approve(id, { by });++ if (out) console.log(`${id}: queued`); else { console.error(`${id}: is ${j.status}, not a proposal`); exit = 1; }++ } else {++ const out = queue.cancel(id, { by, reason: arg('reason', null) });++ if (out.finished) { console.error(`${id}: already ${out.job.status}`); exit = 1; } else console.log(`${id}: ${out.pending ? 'cancel requested (running)' : 'cancelled'}`);++ }++ }++ } else usage(cmd ? `unknown command ${cmd}` : null);++ db.close();++ process.exit(exit);++})().catch((err) => { console.error(err); process.exit(1); });nnew file mode 100644@@@ -0,0 +1,231 @@++#!/usr/bin/env node++'use strict';++/**++ * R2 eviction drill: the canonical B2 copy survives losing the R2 hot-cache copy (roadmap W4 exit++ * criterion; docs/object-model.md#r2-eviction-drill). Dry run unless --execute.++ *++ * node scripts/r2-eviction-drill.js (--vod <id> | --pick) [--execute] [--base-url https://openvibe.media]++ * [--no-http] [--max-mb 512] [--out artifact.json] [--json] [--db ./data/media.db]++ *++ * The VOD must be served from R2 today (storage_provider r2: a B2 canonical copy plus an R2 cache copy),++ * not recording, not under a retention hold (a hold freezes placement), and public or unlisted unless++ * --no-http. --pick chooses the smallest such VOD up to --max-mb. Before anything changes, the B2 and R2++ * copies must have the same size (and the vods row's, when it has one) and the same first MiB: evicting++ * the cache in front of a corrupt canonical copy would lose the only good one, so the drill refuses.++ *++ * With --execute, through the storage engine's own tier moves (server/vod/vod-storage.js):++ * 1. before HEAD B2 + R2; GET <base>/v/<id>?raw=1 redirects to R2 and serves the same first MiB++ * 2. evict demoteFromR2(id): B2 canonical checked, the R2 copy deleted, the row flipped to b2++ * 3. from B2 HEAD R2 is 404; GET /v/<id> now redirects to B2, which serves the same first MiB++ * 4. re-warm promoteToR2(id): B2 copied back to R2 (verified by HEAD), the row flipped to r2++ * 5. after GET /v/<id> redirects to R2 again with the same first MiB; media_locations says b2 and r2 present++ * A failed step stops the drill; the VOD then stays where the last successful step left it (after the++ * eviction that is B2, which still serves it). --no-http checks the playback decision in-process++ * (resolvePlayback) instead of GET /v, which otherwise counts as one view from this host.++ *++ * Writes a JSON artifact (--out, default ./data/drills/r2-eviction-<vod>-<time>.json) with every step,++ * the copies' sizes and first-MiB sha256, media_locations before/after, and the verdict.++ * Exit 0 = pass (or a dry run whose checks passed), 1 = a step failed, 2 = refused (preconditions).++ * Never run it against production without the owner's go-ahead.++ */++const fs = require('fs');++const path = require('path');++const crypto = require('crypto');++++const MIB = 1024 * 1024;++++function parseArgs(argv) {++ const args = argv.slice(2);++ const has = (n) => args.includes(`--${n}`);++ const arg = (n, d) => { const i = args.indexOf(`--${n}`); return i >= 0 ? args[i + 1] : d; };++ return {++ vodId: arg('vod') != null ? Number(arg('vod')) : null,++ pick: has('pick'),++ execute: has('execute'),++ http: !has('no-http'),++ baseUrl: arg('base-url', null),++ maxMb: Number(arg('max-mb', 512)),++ out: arg('out', null),++ json: has('json'),++ db: arg('db', null),++ };++}++++/** Which provider a presigned/redirect URL points at (path-style endpoint or bucket.host). */++function providerOfUrl(url, storage) {++ let u;++ try { u = new URL(url); } catch { return null; }++ for (const p of ['r2', 'b2']) {++ const bucket = storage.bucketFor(p);++ const ep = storage.endpointFor ? storage.endpointFor(p) : null;++ if (!bucket) continue;++ const host = ep ? new URL(ep).host : null;++ if (host && u.host === host && u.pathname.startsWith(`/${bucket}/`)) return p;++ if (host && u.host === `${bucket}.${host}`) return p;++ if (!host && (u.host.startsWith(`${bucket}.`) || u.pathname.startsWith(`/${bucket}/`))) return p;++ }++ return null;++}++++async function firstMibSha(url, fetchImpl) {++ const res = await fetchImpl(url, { headers: { Range: `bytes=0-${MIB - 1}` } });++ if (res.status !== 206 && res.status !== 200) throw new Error(`GET ${new URL(url).host}: HTTP ${res.status}`);++ const body = Buffer.from(await res.arrayBuffer());++ const bytes = res.status === 200 ? body.subarray(0, MIB) : body;++ return { status: res.status, sha256: crypto.createHash('sha256').update(bytes).digest('hex'), bytes: bytes.length, content_range: res.headers.get('content-range') };++}++++/**++ * runDrill(opts, deps) -> artifact. deps: { storage (vod-storage), db, model, fetch, now } — injectable++ * so test/r2-eviction-drill.test.js runs it against a fake S3 and a local Media.++ */++async function runDrill(opts, deps) {++ const { storage, db, model } = deps;++ const fetchImpl = deps.fetch || globalThis.fetch;++ const nowIso = () => new Date((deps.now || Date.now)()).toISOString();++ const art = {++ drill: 'r2-eviction', version: 1, mode: opts.execute ? 'execute' : 'dry-run', started_at: nowIso(), finished_at: null,++ vod_id: null, object_id: null, key: null, http: !!opts.http, base_url: opts.baseUrl || null,++ buckets: { b2: storage.bucketFor('b2'), r2: storage.bucketFor('r2') },++ checks: {}, steps: [], locations: {}, verdict: null, reason: null,++ };++ const step = (name, ok, detail = {}) => { art.steps.push({ name, ok, at: nowIso(), ...detail }); return ok; };++ const locations = () => (art.object_id ? model.listLocations(art.object_id).map(l => ({ provider: l.provider, state: l.state, size_bytes: l.size_bytes, verified_at: l.verified_at })) : []);++ const refuse = (reason) => { art.verdict = 'refused'; art.reason = reason; art.finished_at = nowIso(); return art; };++ const fail = (reason) => { art.verdict = 'fail'; art.reason = reason; art.finished_at = nowIso(); art.locations.at_failure = locations(); return art; };++++ if (!storage.providerConfigured('b2') || !storage.providerConfigured('r2')) return refuse('B2 and R2 must both be configured (MEDIA_B2_*, MEDIA_R2_*)');++++ // ── Choose the VOD ──++ let vod = null;++ if (opts.vodId) vod = db.get('SELECT * FROM vods WHERE id = ?', [opts.vodId]);++ else if (opts.pick) {++ const cands = db.all(`SELECT * FROM vods WHERE storage_provider = 'r2' AND COALESCE(is_recording, 0) = 0 AND COALESCE(clips_only, 0) = 0++ AND file_path IS NOT NULL AND COALESCE(file_size, 0) > 0 AND file_size <= ?++ ${opts.http ? "AND COALESCE(visibility, CASE WHEN is_public THEN 'public' ELSE 'private' END) != 'private'" : ''}++ ORDER BY file_size ASC LIMIT 50`, [Math.max(1, opts.maxMb) * MIB]);++ vod = cands.find(v => !model.isHeldRow(v)) || null;++ if (!vod) return refuse(`no VOD served from R2 fits (not recording, not held${opts.http ? ', public or unlisted' : ''}, at most ${opts.maxMb} MB)`);++ } else return refuse('name a VOD (--vod <id>) or --pick one');++ if (!vod) return refuse(`vod ${opts.vodId} not found`);++ art.vod_id = vod.id;++ art.object_id = vod.object_id || null;++ art.key = storage.keyForVod(vod);++ const vis = vod.visibility || (vod.is_public ? 'public' : 'private');++ art.checks.vod = { app_id: vod.app_id, storage_provider: storage.providerOf(vod), is_recording: !!vod.is_recording, visibility: vis, file_size: Number(vod.file_size) || null };++ if (storage.providerOf(vod) !== 'r2') return refuse(`vod ${vod.id} is served from ${storage.providerOf(vod)}, not R2: there is no R2 copy to evict`);++ if (vod.is_recording) return refuse(`vod ${vod.id} is recording`);++ if (vod.clips_only) return refuse(`vod ${vod.id} is a clips-only recording`);++ if (model.isHeldRow(vod)) return refuse(`vod ${vod.id} is under a retention hold (holds freeze placement)`);++ if (opts.http && vis === 'private') return refuse(`vod ${vod.id} is private: /v/<id> would not serve it anonymously (use --no-http)`);++ if (opts.http && !opts.baseUrl) return refuse('--base-url (or MEDIA_PUBLIC_URL) is needed for the HTTP checks');++ art.locations.before = locations();++++ // ── Both copies agree (size, and the first MiB) ──++ const [b2, r2] = await Promise.all([storage.headObject('b2', art.key), storage.headObject('r2', art.key)]);++ art.checks.head_before = { b2, r2 };++ if (!b2) return refuse('the B2 canonical copy is missing: evicting R2 would lose the only copy');++ if (!r2) return refuse('there is no R2 copy (HEAD 404) although the row says r2');++ if (b2.size !== r2.size) return refuse(`B2 (${b2.size}) and R2 (${r2.size}) differ in size: the canonical copy may be corrupt`);++ if (vod.file_size && Number(vod.file_size) !== b2.size) return refuse(`the copies (${b2.size}) differ from the recorded size (${vod.file_size})`);++ const b2Loc = (art.locations.before || []).find(l => l.provider === 'b2');++ if (b2Loc && ['corrupt', 'missing'].includes(b2Loc.state)) return refuse(`media_locations marks the B2 copy ${b2Loc.state}`);++ const [shaB2, shaR2] = await Promise.all([++ firstMibSha(await storage.presignGet('b2', art.key, 600), fetchImpl),++ firstMibSha(await storage.presignGet('r2', art.key, 600), fetchImpl),++ ]);++ art.checks.first_mib = { b2: shaB2.sha256, r2: shaR2.sha256 };++ if (shaB2.sha256 !== shaR2.sha256) return refuse('B2 and R2 differ in their first MiB: the canonical copy may be corrupt');++ const expectSha = shaR2.sha256;++++ /** Where playback goes now: GET /v/<id>?raw=1 (or resolvePlayback in-process) and the first MiB served there. */++ const served = async () => {++ if (!opts.http) {++ const plan = await storage.resolvePlayback(db.get('SELECT * FROM vods WHERE id = ?', [vod.id]));++ if (!plan || plan.kind !== 'redirect') return { provider: plan ? plan.kind : null };++ const got = await firstMibSha(plan.url, fetchImpl);++ return { via: 'resolvePlayback', provider: plan.provider, status: 302, first_mib_sha256: got.sha256 };++ }++ const url = `${String(opts.baseUrl).replace(/\/+$/, '')}/v/${vod.id}?raw=1`;++ const res = await fetchImpl(url, { redirect: 'manual', headers: { Accept: 'video/*' } });++ const location = res.headers.get('location');++ if (res.status !== 302 || !location) return { via: 'http', status: res.status, provider: null };++ const got = await firstMibSha(location, fetchImpl);++ return { via: 'http', url, status: 302, provider: providerOfUrl(location, storage), host: new URL(location).host, range_status: got.status, first_mib_sha256: got.sha256 };++ };++++ if (!opts.execute) {++ const now = opts.http ? null : await served().catch(e => ({ error: e.message }));++ step('plan', true, {++ would: ['evict: demoteFromR2 (B2 canonical checked, R2 copy deleted, row -> b2)', 'prove: /v/<id> redirects to B2 and serves the same first MiB',++ 're-warm: promoteToR2 (B2 copied to R2, row -> r2)', 'prove: /v/<id> redirects to R2 again'],++ served_now: opts.http ? 'not requested in a dry run (GET /v counts a view)' : now,++ size_bytes: b2.size,++ });++ art.verdict = 'dry-run';++ art.finished_at = nowIso();++ return art;++ }++++ // 1. before++ const before = await served();++ if (!step('before: served from R2', before.provider === 'r2' && before.first_mib_sha256 === expectSha, before)) return fail('playback is not coming from R2 before the eviction');++ // 2. evict++ const ev = await storage.demoteFromR2(vod.id);++ const rowAfterEvict = db.get('SELECT storage_provider FROM vods WHERE id = ?', [vod.id]);++ const r2Gone = await storage.headObject('r2', art.key);++ art.locations.after_evict = locations();++ if (!step('evict: R2 copy removed', !!(ev && ev.ok) && !r2Gone && rowAfterEvict.storage_provider === 'b2', { result: ev, r2_head: r2Gone, storage_provider: rowAfterEvict.storage_provider })) {++ return fail('the eviction did not complete');++ }++ // 3. served from B2++ const fromB2 = await served();++ const b2Head = await storage.headObject('b2', art.key);++ if (!step('from B2: served from the canonical copy', fromB2.provider === 'b2' && fromB2.first_mib_sha256 === expectSha && !!b2Head && b2Head.size === b2.size, { ...fromB2, b2_head: b2Head })) {++ return fail('after the eviction the VOD is not served from B2');++ }++ // 4. re-warm++ const pr = await storage.promoteToR2(vod.id);++ const r2Back = await storage.headObject('r2', art.key);++ const rowAfterWarm = db.get('SELECT storage_provider FROM vods WHERE id = ?', [vod.id]);++ if (!step('re-warm: R2 copy restored from B2', !!(pr && pr.ok) && !!r2Back && r2Back.size === b2.size && rowAfterWarm.storage_provider === 'r2', { result: pr, r2_head: r2Back, storage_provider: rowAfterWarm.storage_provider })) {++ return fail('the re-warm did not complete (the VOD is still served from B2)');++ }++ // 5. after++ const after = await served();++ art.locations.after = locations();++ const locOk = !art.object_id || ['b2', 'r2'].every(p => (art.locations.after.find(l => l.provider === p) || {}).state === 'present');++ if (!step('after: served from R2 again', after.provider === 'r2' && after.first_mib_sha256 === expectSha && locOk, { ...after, locations_present: locOk })) {++ return fail('after the re-warm the VOD is not served from R2');++ }++ art.verdict = 'pass';++ art.finished_at = nowIso();++ return art;++}++++function summarize(art) {++ const lines = [`R2 eviction drill (${art.mode}) vod ${art.vod_id ?? '-'} ${art.key || ''}: ${art.verdict}${art.reason ? ` — ${art.reason}` : ''}`];++ for (const s of art.steps) lines.push(` ${s.ok ? 'ok ' : 'FAIL'} ${s.name}`);++ return lines.join('\n');++}++++if (require.main === module) {++ const opts = parseArgs(process.argv);++ if (opts.db) process.env.DB_PATH = path.resolve(opts.db);++ const config = require('../server/config');++ if (!opts.baseUrl) opts.baseUrl = config.publicUrl;++ const storage = require('../server/vod/vod-storage');++ const db = require('../server/db/database');++ const model = require('../server/objects/model');++ (async () => {++ const art = await runDrill(opts, { storage, db, model });++ const out = opts.out || path.join(path.dirname(config.db.path), 'drills', `r2-eviction-${art.vod_id ?? 'none'}-${art.started_at.replace(/[:.]/g, '-')}.json`);++ fs.mkdirSync(path.dirname(out), { recursive: true });++ fs.writeFileSync(out, JSON.stringify(art, null, 2));++ console.log(opts.json ? JSON.stringify(art, null, 2) : `${summarize(art)}\nartifact: ${out}`);++ db.close();++ process.exit(art.verdict === 'pass' || art.verdict === 'dry-run' ? 0 : art.verdict === 'refused' ? 2 : 1);++ })().catch((err) => { console.error(err); process.exit(1); });++}++++module.exports = { runDrill, providerOfUrl, summarize, parseArgs };@@@ -342,7 +342,7 @@ function tenantCors(req, res, next) {if (origin && req.appRow && db.appAllowedOrigins(req.appRow).includes(origin)) {res.set('Access-Control-Allow-Origin', origin);res.set('Vary', 'Origin');-- res.set('Access-Control-Allow-Headers', 'Authorization, Content-Type');++ res.set('Access-Control-Allow-Headers', 'Authorization, Content-Type, X-Upload-Token, X-Content-SHA256, Idempotency-Key');res.set('Access-Control-Allow-Methods', 'GET, POST, PUT, DELETE, OPTIONS');}next();@@@ -55,6 +55,27 @@ const config = {publicMaxMb: intEnv('MEDIA_PUBLIC_OBJECT_MAX_MB', 500),publicTargetMb: intEnv('MEDIA_PUBLIC_OBJECT_TARGET_MB', 256),publicWarnMb: intEnv('MEDIA_PUBLIC_OBJECT_WARN_MB', 384),++ // Multipart uploads (server/objects/multipart.js): parts are stored locally and assembled at complete.++ multipartMaxMb: intEnv('MEDIA_MULTIPART_MAX_MB', 20480), // largest object a multipart upload may declare++ multipartMinPartMb: intEnv('MEDIA_MULTIPART_MIN_PART_MB', 5), // every part but the last is at least this++ multipartMaxPartMb: intEnv('MEDIA_MULTIPART_MAX_PART_MB', 256),++ multipartDefaultPartMb: intEnv('MEDIA_MULTIPART_PART_MB', 64),++ multipartTtlHours: intEnv('MEDIA_MULTIPART_TTL_HOURS', 24), // an unfinished session's parts are purged after this++ uploadMinFreeMb: intEnv('MEDIA_UPLOAD_MIN_FREE_MB', 10240), // multipart needs 2x its size free plus this++ },++++ // Job system (server/jobs/; docs/object-model.md#jobs).++ jobs: {++ enabled: !['0', 'false', 'off'].includes(String(process.env.MEDIA_JOBS_ENABLED || '').toLowerCase()),++ pollMs: intEnv('MEDIA_JOBS_POLL_MS', 5000),++ lightConcurrency: Math.max(1, intEnv('MEDIA_JOBS_LIGHT_CONCURRENCY', 2)), // thumbnails, scans++ heavyConcurrency: Math.max(1, intEnv('MEDIA_JOBS_HEAVY_CONCURRENCY', 1)), // split, remux++ // Heavy jobs wait while a recording is running (as the health job does), unless this is on.++ heavyWhileRecording: ['1', 'true', 'on'].includes(String(process.env.MEDIA_JOBS_HEAVY_WHILE_RECORDING || '').toLowerCase()),++ leaseS: Math.max(30, intEnv('MEDIA_JOBS_LEASE_S', 120)),++ // The size-invariant validator runs this often (hours; 0 = only on demand). It proposes jobs, never runs them.++ invariantScanHours: intEnv('MEDIA_INVARIANT_SCAN_HOURS', 24),++ retentionDays: intEnv('MEDIA_JOBS_RETENTION_DAYS', 30), // finished thumbnail jobs are pruned after this},// Scheduled copy verification (server/objects/verify-job.js; docs/object-model.md#scheduled-verification).@@@ -28,6 +28,7 @@ function getDb() {const schema = fs.readFileSync(path.join(__dirname, 'schema.sql'), 'utf8');database.exec(schema);migrateColumns();++ migrateJobsTable(schema);ensureObjectTriggers();normalizeThumbnailUrls();seedSettings();@@@ -89,6 +90,35 @@ function migrateColumns() {}}++/**++ * media_jobs was created schema-only (integer id, no app_id, status 'done') before the job worker++ * existed; nothing ever wrote it. Rebuild that shape into the current one from schema.sql, keeping++ * any rows (id mjob_legacy_<n>, tenant from the object, done -> succeeded), then create the indexes++ * that need the new columns. Idempotent: a database already in the new shape only gets its indexes.++ */++function migrateJobsTable(schema) {++ const cols = database.prepare('PRAGMA table_info(media_jobs)').all().map(c => c.name);++ if (!cols.includes('app_id')) {++ const ddl = /CREATE TABLE IF NOT EXISTS media_jobs \([\s\S]*?\n\);/.exec(schema);++ if (!ddl) throw new Error('schema.sql has no media_jobs table');++ database.transaction(() => {++ database.exec('ALTER TABLE media_jobs RENAME TO media_jobs_v0');++ database.exec(ddl[0]);++ const moved = database.prepare(`INSERT INTO media_jobs (id, app_id, object_id, job_type, status, attempts, checkpoint, error, created_at, updated_at)++ SELECT 'mjob_legacy_' || v.id, COALESCE((SELECT o.app_id FROM media_objects o WHERE o.id = v.object_id), 'unknown'), v.object_id, v.job_type,++ CASE v.status WHEN 'done' THEN 'succeeded' ELSE v.status END, v.attempts, v.checkpoint, v.error, v.created_at, v.updated_at++ FROM media_jobs_v0 v`).run().changes;++ database.exec('DROP TABLE media_jobs_v0');++ console.log(`[DB] Rebuilt media_jobs for the job worker (${moved} row(s) kept)`);++ })();++ }++ database.exec(`++ CREATE INDEX IF NOT EXISTS idx_media_jobs_status ON media_jobs(status, job_type);++ CREATE INDEX IF NOT EXISTS idx_media_jobs_app ON media_jobs(app_id, id);++ CREATE INDEX IF NOT EXISTS idx_media_jobs_object ON media_jobs(object_id, job_type, status);++ CREATE UNIQUE INDEX IF NOT EXISTS idx_media_jobs_idem ON media_jobs(app_id, idempotency_key) WHERE idempotency_key IS NOT NULL;`);++}++/*** Object-model guards that must hold on EVERY delete path, including the many* inherited ones that DELETE a vods/clips/files/pastes row directly:@@@ -275,20 +275,65 @@ CREATE TABLE IF NOT EXISTS media_variants (UNIQUE(object_id, variant_name));++-- Generic derivative/maintenance jobs (server/jobs/; docs/object-model.md#jobs). A database made before++-- the worker existed has the schema-only shape (integer id, no app_id); database.js migrateJobsTable()++-- rebuilds it into this one. The other indexes are created there too, after the rebuild.CREATE TABLE IF NOT EXISTS media_jobs (-- id INTEGER PRIMARY KEY AUTOINCREMENT,-- object_id TEXT,-- job_type TEXT NOT NULL,-- status TEXT NOT NULL DEFAULT 'queued' CHECK(status IN ('queued', 'running', 'done', 'failed', 'cancelled')),++ id TEXT PRIMARY KEY, -- mjob_<ULID>++ app_id TEXT NOT NULL, -- the tenant that owns the job++ object_id TEXT, -- the object it works on (NULL for tenant-wide jobs: invariant.scan)++ job_type TEXT NOT NULL, -- thumbnail.regenerate | invariant.scan | object.split | object.remux++ status TEXT NOT NULL DEFAULT 'queued' CHECK(status IN ('proposed', 'queued', 'running', 'succeeded', 'failed', 'cancelled')),++ idempotency_key TEXT, -- unique per tenant: a repeat answers with the same job++ request_hash TEXT, -- sha256 of { type, object_id, params }: a different request under a used key is refused++ params TEXT NOT NULL DEFAULT '{}',++ result TEXT,attempts INTEGER NOT NULL DEFAULT 0,-- checkpoint TEXT,++ max_attempts INTEGER NOT NULL DEFAULT 3,++ run_after DATETIME, -- queued: not before (retry backoff)++ lease_until DATETIME, -- running: renewed while the worker holds the job++ checkpoint TEXT, -- handler progress (JSON); a retry resumes from iterror TEXT,++ error_code TEXT, -- stable code of the last failure (media_unavailable, quota_exceeded, …)++ cancel_requested INTEGER NOT NULL DEFAULT 0,++ created_by TEXT, -- svc:<id> | app:<app> | app:<app>:user:<id> | system:<what>++ owner_user_id INTEGER, -- the app's user it was created for (X-OV-User-Id), if any++ decided_by TEXT, -- who approved or rejected a proposal++ decided_at DATETIME,created_at DATETIME DEFAULT CURRENT_TIMESTAMP,-- updated_at DATETIME DEFAULT CURRENT_TIMESTAMP++ updated_at DATETIME DEFAULT CURRENT_TIMESTAMP,++ started_at DATETIME,++ finished_at DATETIME);CREATE INDEX IF NOT EXISTS idx_media_jobs_status ON media_jobs(status, job_type);++-- Multipart uploads of native v2 objects (server/objects/multipart.js). Parts are stored under++-- OBJECTS_PATH/.parts/<upload id>/ and assembled at complete; one active session per object.++CREATE TABLE IF NOT EXISTS media_uploads (++ id TEXT PRIMARY KEY, -- mup_<ULID>++ object_id TEXT NOT NULL,++ app_id TEXT NOT NULL,++ part_size INTEGER NOT NULL,++ total_size INTEGER NOT NULL,++ parts_expected INTEGER NOT NULL,++ status TEXT NOT NULL DEFAULT 'active' CHECK(status IN ('active', 'completing', 'completed', 'aborted', 'expired')),++ created_at DATETIME DEFAULT CURRENT_TIMESTAMP,++ updated_at DATETIME DEFAULT CURRENT_TIMESTAMP,++ expires_at DATETIME NOT NULL,++ completed_at DATETIME++);++CREATE INDEX IF NOT EXISTS idx_media_uploads_object ON media_uploads(object_id, status);++CREATE INDEX IF NOT EXISTS idx_media_uploads_expiry ON media_uploads(status, expires_at);++++CREATE TABLE IF NOT EXISTS media_upload_parts (++ upload_id TEXT NOT NULL,++ part_number INTEGER NOT NULL, -- 1-based++ size_bytes INTEGER NOT NULL,++ sha256 TEXT NOT NULL,++ received_at DATETIME DEFAULT CURRENT_TIMESTAMP,++ PRIMARY KEY (upload_id, part_number)++);++-- Retention holds: an object with an unreleased hold cannot be deleted or moved between tiers.CREATE TABLE IF NOT EXISTS media_holds (id INTEGER PRIMARY KEY AUTOINCREMENT,@@@ -9,6 +9,8 @@* clip.ready | clip.failed → media.clip.ready | media.clip.failed (subject clip <id>)* media.object.uploaded → media.object.uploaded (subject object <id>)* storage.alert | storage.recovered → media.storage.alert | media.storage.recovered++ * job state changes (server/jobs/queue.js) → media.job.proposed | queued | started | retrying |++ * succeeded | failed | cancelled (subject job <id>; Events only, no webhook)** The outbox row is written INSIDE the SQLite transaction that makes the state change it describes* (record(), called from webhooks.announce()): the event exists if and only if the change committed,@@@ -36,6 +38,12 @@ const TYPES = {'storage.recovered': ['media.storage.recovered', 'storage'],};++// media.job.<transition>: progress is low priority, the outcome (and a proposal waiting for its owner) important.++const JOB_TRANSITIONS = {++ proposed: 'important', queued: 'low', started: 'low', retrying: 'low',++ succeeded: 'important', failed: 'important', cancelled: 'important',++};++let outbox = null;const stats = { queued: 0, lastError: null };@@@ -80,10 +88,6 @@ function slim(appId, data) {* or the webhook event has no durable twin.*/function record(webhookEvent, appId, data) {-- if (!outbox) return null;-- // Developer-project sandbox tenants (ADR-014) produce no platform events: sandbox activity must-- // never reach production consumers.-- if (appId && db.isSandboxTenant(appId)) return null;const map = TYPES[webhookEvent];if (!map) return null;const [eventType, subjectType] = map;@@@ -91,18 +95,51 @@ function record(webhookEvent, appId, data) {const subject = subjectType === 'storage'? { type: 'storage', id: String((data && data.kind) || 'storage') }: { type: subjectType, id: String(id == null ? 'unknown' : id) };++ return enqueue(eventType, appId, subject, data, eventType === 'media.storage.recovered' ? 'low' : 'important');++}++++/**++ * Queue media.job.<transition> for a job row (queue.jobPublic shape). Same rule as record(): call it++ * inside the transaction that changes the job's state; it throws when the insert fails.++ */++function recordJob(transition, job) {++ const priority = JOB_TRANSITIONS[transition];++ if (!priority) throw new Error(`unknown job transition ${transition}`);++ return enqueue(`media.job.${transition}`, job.app_id, { type: 'job', id: String(job.id) }, job, priority);++}++++function enqueue(eventType, appId, subject, data, priority) {++ if (!outbox) return null;++ // Developer-project sandbox tenants (ADR-014) produce no platform events: sandbox activity must++ // never reach production consumers.++ if (appId && db.isSandboxTenant(appId)) return null;const env = outbox.enqueue({event_type: eventType,actor: { type: 'service', id: 'media' },subject,visibility: 'internal',-- priority: eventType === 'media.storage.recovered' ? 'low' : 'important',++ priority,payload: slim(appId, data),});stats.queued++;return env;}++/**++ * Operator scripts (scripts/media-jobs.js) change job state in the same database as the running++ * service. When the service's outbox table exists (the outbox is on there), the script writes its++ * events into it too, and the service's relay publishes them. No relay runs in the script.++ */++function initWriter() {++ if (outbox) return outbox;++ const raw = db.getDb();++ if (!raw.prepare("SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = 'event_outbox'").get()) return null;++ const client = createClient({ baseUrls: { events: 'http://127.0.0.1:9' }, retries: 0 });++ outbox = createOutbox(raw, { events: createEventsClient(client, { source: 'media' }) });++ outbox.ensureSchema();++ return outbox;++}++/** Wake the relay once the transaction that queued events has committed. */function kick() {if (outbox) setImmediate(() => outbox && outbox.kick());@@@ -115,4 +152,4 @@ function status() {function _reset() { if (outbox) outbox.stop(); outbox = null; stats.queued = 0; stats.lastError = null; }--module.exports = { init, record, kick, status, TYPES, _reset };++module.exports = { init, initWriter, record, recordJob, kick, status, TYPES, JOB_TRANSITIONS, _reset };@@@ -33,6 +33,7 @@ const observability = require('./observability');const instrumented = observability.instrument(app, { release: release.release });// Object API v2 content uploads read the raw request body, so they sit ahead of the body parsers.app.put('/api/v2/:app/objects/:id/content', ...require('./objects/routes').contentHandlers);++app.put('/api/v2/:app/objects/:id/multipart/:uploadId/parts/:n', ...require('./objects/routes').partHandlers);app.use(express.json({ limit: '2mb' }));// ── Visitor sign-in (OAuth client `media` on the Network; same module as Community/Tools) ──@@@ -78,7 +79,7 @@ app.options(['/api/v1/:app/*', '/api/v2/:app/*'], (req, res) => {if (origin && appRow && db.appAllowedOrigins(appRow).includes(origin)) {res.set('Access-Control-Allow-Origin', origin);res.set('Vary', 'Origin');-- res.set('Access-Control-Allow-Headers', 'Authorization, Content-Type');++ res.set('Access-Control-Allow-Headers', 'Authorization, Content-Type, X-Upload-Token, X-Content-SHA256, Idempotency-Key');res.set('Access-Control-Allow-Methods', 'GET, POST, PUT, DELETE, OPTIONS');}res.sendStatus(204);@@@ -143,6 +144,7 @@ app.use('/api/v1/:app/thumbnails', require('./thumbnails/routes'));app.use('/api/v1/:app/assets', require('./assets/routes'));app.use('/api/v1/:app/admin/storage', require('./admin/routes'));app.use('/api/v2/:app/objects', require('./objects/routes')); // canonical object API (docs/object-model.md)++app.use('/api/v2/:app/jobs', require('./jobs/routes')); // media jobs: thumbnails, split/remux, invariant scansapp.use('/o', require('./objects/routes').publicRouter); // object bytes (public, or signed)app.use('/', require('./public/routes')); // /v /c /p /t /f@@@ -173,6 +175,7 @@ vodStorage.start(); // tiering sweephealthJob.start(); // health scan + quarantine cleanuprequire('./vod/clip-jobs').start(); // failed clip re-cuts (auto-retry, hot-fetch)require('./objects/verify-job').start(); // scheduled copy verification (bounded batches; never deletes)++require('./jobs/worker').start(); // media_jobs worker (light + heavy lanes; proposals wait for their owner)// Descriptor watchdog: a leak here once pinned 80 GB of deleted recordings to the disk.every(5 * 60 * 1000, () => {try {@@@ -185,6 +188,8 @@ every(60 * 60 * 1000, () => thumbService.cleanupOldThumbnails()); // stale live// Object model: purge native objects whose soft-delete retention has passed (held ones are kept).every(60 * 60 * 1000, () => {try { const n = require('./objects/model').purgeExpired(); if (n) console.log(`[Objects] Purged ${n} expired deleted object(s)`); } catch (err) { console.warn('[Objects] purge:', err.message); }++ // Incomplete multipart uploads past MEDIA_MULTIPART_TTL_HOURS: their parts are deleted (the objects stay uploading).++ try { const r = require('./objects/multipart').purgeExpired(); if (r.expired || r.orphan_dirs) console.log(`[Objects] Multipart: ${r.expired} expired session(s), ${r.orphan_dirs} orphan part dir(s) removed`); } catch (err) { console.warn('[Objects] multipart purge:', err.message); }});// Project rows that have no media_object yet (first boot after the upgrade: all of them).setTimeout(() => {@@@ -229,6 +234,7 @@ function shutdown(signal) {try { vodStorage.stop(); } catch { /* */ }try { healthJob.stop(); } catch { /* */ }try { require('./objects/verify-job').stop(); } catch { /* */ }++ try { require('./jobs/worker').stop(); } catch { /* */ }try { auth.stopJwksRefresh(); } catch { /* */ }try { require('./events')._reset(); } catch { /* */ }for (const t of timers) clearInterval(t);nnew file mode 100644@@@ -0,0 +1,310 @@++/**++ * OpenVibe.Media — job types object.split and object.remux (heavy lane)++ *++ * Both make NEW objects from a ready vod/clip (or a native video/audio object) with a stream copy++ * (ffmpeg -c copy: no re-encode). They never touch the source: its bytes, locations, visibility and++ * lifecycle stay as they are. Each output is a native object in the same tenant with the source's++ * owner, visibility `private` (the owner decides what becomes public), a verified local copy, and a++ * `derived_from` relationship to the source ({ job_id, method, part, parts, start_seconds, … }).++ *++ * object.split params { parts | segment_seconds, … } one object per part, cut at keyframes (a part++ * can start up to one keyframe interval early, so neighbours may overlap slightly).++ * Progress is checkpointed per part: a retry resumes after the last finished part.++ * result { source_id, parts: [{ part, object_id, start_seconds, duration_seconds, size_bytes }],++ * oversized_parts, method: 'split' }++ * object.remux params {} one object: the whole source remuxed (seek index,++ * duration; MP4 +faststart). Also the source's `remux` variant.++ * result { source_id, object_id, size_bytes, duration_seconds, method: 'remux' }++ *++ * Refused before any work: a source that is not ready (permanent), a tenant quota the output would++ * exceed (permanent), less free disk than the source size plus MEDIA_UPLOAD_MIN_FREE_MB (retried later).++ */++'use strict';++++const fs = require('fs');++const path = require('path');++const crypto = require('crypto');++const { spawn } = require('child_process');++const config = require('../config');++const db = require('../db/database');++const model = require('../objects/model');++const { JobError } = require('./queue');++++const MB = 1024 * 1024;++const MAX_PARTS = 1000;++const MEDIA_KINDS = ['vod', 'clip'];++++function isMediaObject(obj) {++ return !!obj && (MEDIA_KINDS.includes(obj.kind) || /^(video|audio)\//.test(String(obj.mime_type || '')));++}++++function extFor(obj, sourcePath) {++ const fromPath = path.extname(String(sourcePath || obj.canonical_key || '')).toLowerCase();++ if (['.webm', '.mp4', '.mkv', '.mov', '.m4a', '.ogg', '.mp3'].includes(fromPath)) return fromPath === '.mov' ? '.mp4' : fromPath;++ const m = String(obj.mime_type || '');++ if (m === 'video/mp4' || m === 'audio/mp4') return '.mp4';++ if (m === 'video/x-matroska') return '.mkv';++ if (m === 'audio/mpeg') return '.mp3';++ if (m === 'audio/ogg' || m === 'video/ogg') return '.ogg';++ return '.webm';++}++++function mimeFor(ext, obj) {++ return { '.webm': 'video/webm', '.mp4': 'video/mp4', '.mkv': 'video/x-matroska', '.m4a': 'audio/mp4', '.ogg': 'video/ogg', '.mp3': 'audio/mpeg' }[ext]++ || obj.mime_type || 'application/octet-stream';++}++++function existing(p) {++ try { return p && fs.statSync(p).isFile() ? p : null; } catch { return null; }++}++++/**++ * Something ffmpeg can read for this object: a local path, or a presigned R2/B2 URL.++ * Projected vods/clips resolve the way playback does; native objects through their locations.++ */++async function resolveSource(obj) {++ const vodStorage = require('../vod/vod-storage');++ const ref = model.parseLegacyRef(obj.legacy_ref);++ if (ref && (ref.kind === 'vod' || ref.kind === 'clip')) {++ const row = db.get(`SELECT * FROM ${ref.kind === 'vod' ? 'vods' : 'clips'} WHERE id = ?`, [Number(ref.id)]);++ if (!row || !row.file_path) return null;++ const local = ref.kind === 'vod'++ ? existing(vodStorage.localPathForVod(row)) || existing(row.file_path)++ : existing(row.file_path) || existing(path.join(path.resolve(config.vod.clipsPath), path.basename(row.file_path)));++ if (local) return { input: local, name: row.file_path };++ const src = await vodStorage.resolveMediaSource(row);++ return src ? { input: src.value, name: row.file_path, remote: src.kind === 'url' } : null;++ }++ const locs = model.listLocations(obj.id);++ const local = locs.find(l => l.provider === 'local' && l.state !== 'missing' && existing(l.key));++ if (local) return { input: local.key, name: obj.canonical_key || local.key };++ for (const p of ['r2', 'b2']) {++ const l = locs.find(x => x.provider === p && !['missing', 'corrupt'].includes(x.state));++ if (!l || !vodStorage.providerConfigured(p)) continue;++ const url = await vodStorage.presignGet(p, l.key, 6 * 3600).catch(() => null);++ if (url) return { input: url, name: l.key, remote: true };++ }++ return null;++}++++function freeBytes(dir) {++ try { const st = fs.statfsSync(dir); return Number(st.bavail) * Number(st.bsize); } catch { return Infinity; }++}++++function sha256File(p) {++ return new Promise((resolve, reject) => {++ const h = crypto.createHash('sha256');++ const s = fs.createReadStream(p);++ s.on('data', (c) => h.update(c));++ s.on('error', reject);++ s.on('end', () => resolve(h.digest('hex')));++ });++}++++/** ffmpeg with a deadline and the job's abort signal. Resolves { ok, code, stderr }. */++function ffmpeg(args, { signal, timeoutMs }) {++ return new Promise((resolve) => {++ let proc;++ try { proc = spawn('ffmpeg', args, { stdio: ['ignore', 'ignore', 'pipe'] }); } catch (err) { return resolve({ ok: false, code: -1, stderr: err.message }); }++ let stderr = '';++ let done = false;++ const finish = (r) => { if (done) return; done = true; clearTimeout(t); if (signal) signal.removeEventListener('abort', onAbort); resolve(r); };++ const kill = () => { try { proc.kill('SIGKILL'); } catch { /* gone */ } };++ const onAbort = () => kill();++ const t = setTimeout(() => { stderr += '\n[timed out]'; kill(); }, timeoutMs);++ if (signal) { if (signal.aborted) kill(); else signal.addEventListener('abort', onAbort, { once: true }); }++ proc.stderr.on('data', (d) => { stderr = (stderr + d).slice(-4000); });++ proc.on('error', (err) => finish({ ok: false, code: -1, stderr: err.message }));++ proc.on('close', (code) => finish({ ok: code === 0, code, stderr }));++ });++}++++function inputArgs(src) {++ return src.remote ? ['-reconnect', '1', '-reconnect_streamed', '1', '-reconnect_delay_max', '10'] : [];++}++++/** Budget for one stream copy of `bytes`: ~45 s per GB, at least 3 min, at most 2 h. */++function budgetMs(bytes) {++ return Math.min(2 * 3600 * 1000, Math.max(180000, Math.round((Number(bytes) || 0) / 1e9 * 45000) + 60000));++}++++function probeDuration(p) {++ return require('../vod/media-tools').probeVodInfo(p).then(i => i.duration || 0).catch(() => 0);++}++++function loadSource(job) {++ const obj = model.getObject(job.object_id);++ if (!obj || obj.app_id !== job.app_id) throw new JobError('not_found', 'The source object no longer exists', { permanent: true });++ if (obj.lifecycle_status !== 'ready') throw new JobError('not_ready', `The source object is ${obj.lifecycle_status}`, { permanent: true });++ if (!isMediaObject(obj)) throw new JobError('media.job.invalid', 'Only vod/clip objects (or video/audio objects) can be split or remuxed', { permanent: true });++ return obj;++}++++function checkRoom(obj, needBytes) {++ const reserve = config.objects.uploadMinFreeMb * MB;++ const free = freeBytes(config.objects.path);++ if (free < needBytes + reserve) {++ throw new JobError('insufficient_disk', `Not enough free disk: need ${Math.ceil((needBytes + reserve) / MB)} MB, ${Math.floor(free / MB)} MB free`, { retryAfterS: 1800 });++ }++ const app = db.getApp(obj.app_id);++ const quota = Number(app && app.quota_bytes) || 0;++ if (quota && model.usedBytes(obj.app_id) + needBytes > quota) {++ throw new JobError('quota_exceeded', 'The tenant storage quota would be exceeded', { permanent: true });++ }++}++++function workDir(jobId) {++ const d = path.join(config.objects.path, '.jobs', jobId);++ fs.mkdirSync(d, { recursive: true });++ return d;++}++++function cleanupWork(jobId) {++ try { fs.rmSync(path.join(config.objects.path, '.jobs', jobId), { recursive: true, force: true }); } catch { /* best effort */ }++}++++/**++ * Register one finished output file as a private native object derived from `src`: create the row,++ * move the bytes under OBJECTS_PATH/<app>/<id>, verified local location, relationship; `alsoInTx`++ * (the job checkpoint) commits with it. Returns the new object id.++ */++async function adopt({ src, file, ext, job, relation, metadata, variant, saveCheckpoint, checkpointFor }) {++ const size = fs.statSync(file).size;++ const hash = await sha256File(file);++ const md = model.parseJson(src.metadata, {});++ const id = require('openvibe-contracts').ids.newId('media');++ const dest = path.join(config.objects.path, src.app_id, id);++ fs.mkdirSync(path.dirname(dest), { recursive: true });++ fs.renameSync(file, dest);++ try {++ saveCheckpoint(checkpointFor(id, size), () => {++ db.run(`INSERT INTO media_objects (id, app_id, namespace, kind, owner_subject, owner_app, owner_user_id, visibility, lifecycle_status,++ mime_type, size_bytes, content_hash, canonical_provider, canonical_key, legacy_ref, metadata)++ VALUES (?, ?, ?, ?, ?, ?, ?, 'private', 'ready', ?, ?, ?, 'local', ?, NULL, ?)`,++ [id, src.app_id, src.namespace || src.app_id, src.kind, src.owner_subject || null, src.owner_app || src.app_id, src.owner_user_id ?? null,++ mimeFor(ext, src), size, hash, dest,++ JSON.stringify({ title: md.title || null, derived_from: src.id, job_id: job.id, filename: `${src.id}${metadata.part ? `-part${metadata.part}` : '-remux'}${ext}`, ...metadata })]);++ model.upsertLocation(id, { provider: 'local', key: dest, state: 'present', size_bytes: size, checksum: hash, verified: true });++ model.setRelationship(id, 'derived_from', src.id, { job_id: job.id, ...relation });++ if (variant) model.setVariant(src.id, variant, id, `${job.type}@1`);++ });++ } catch (err) {++ try { fs.unlinkSync(dest); } catch { /* not moved */ }++ throw err;++ }++ return { id, size };++}++++// ── object.split ─────────────────────────────────────────────++++function validateSplit({ obj, params }) {++ if (!obj) throw new JobError('media.job.invalid', 'object.split needs object_id', { permanent: true });++ if (!isMediaObject(obj)) throw new JobError('media.job.invalid', 'Only vod/clip objects (or video/audio objects) can be split', { permanent: true });++ const p = { ...(params || {}) };++ const parts = p.parts != null ? Number(p.parts) : null;++ const seg = p.segment_seconds != null ? Number(p.segment_seconds) : null;++ if (parts == null && seg == null) throw new JobError('media.job.invalid', 'object.split needs params.parts (2-1000) or params.segment_seconds', { permanent: true });++ if (parts != null && (!Number.isInteger(parts) || parts < 2 || parts > MAX_PARTS)) throw new JobError('media.job.invalid', 'params.parts must be an integer from 2 to 1000', { permanent: true });++ if (seg != null && (!Number.isFinite(seg) || seg < 1)) throw new JobError('media.job.invalid', 'params.segment_seconds must be at least 1', { permanent: true });++ return p;++}++++async function runSplit(job, ctx) {++ const src = loadSource(job);++ const source = await resolveSource(src);++ if (!source) throw new JobError('media_unavailable', 'The source bytes are unavailable (no local file and no cloud copy)', { permanent: true });++ const md = model.parseJson(src.metadata, {});++ const duration = Number(md.duration_seconds) || await probeDuration(source.input);++ if (!(duration > 0)) throw new JobError('duration_unknown', 'The source duration is unknown: remux it first (object.remux)', { permanent: true });++ const parts = job.params.segment_seconds++ ? Math.min(MAX_PARTS, Math.ceil(duration / Number(job.params.segment_seconds)))++ : Math.min(MAX_PARTS, Number(job.params.parts));++ const seg = job.params.segment_seconds ? Number(job.params.segment_seconds) : Math.ceil(duration / parts);++ const cp = ctx.checkpoint && Array.isArray(ctx.checkpoint.parts) ? ctx.checkpoint : { parts: [] };++ const doneParts = new Set(cp.parts.map(p => p.part));++ const remaining = Math.max(0, (Number(src.size_bytes) || 0) - cp.parts.reduce((a, p) => a + p.size_bytes, 0));++ checkRoom(src, remaining);++ const ext = extFor(src, source.name);++ const dir = workDir(job.id);++ const maxBytes = Number(job.params.max_bytes) || require('../objects/invariant').thresholds().maxBytes;++++ for (let i = 0; i < parts; i++) {++ const part = i + 1;++ const start = i * seg;++ if (start >= duration) break;++ if (doneParts.has(part)) continue;++ if (ctx.signal.aborted) throw ctx.signal.reason || new Error('aborted');++ const len = Math.min(seg, duration - start);++ const out = path.join(dir, `part-${part}${ext}`);++ // A cloud source is presigned again for every part: a long split outlives one presigned URL.++ const src2 = part === 1 || !source.remote ? source : (await resolveSource(src)) || source;++ const r = await ffmpeg(['-y', '-nostdin', '-v', 'error', ...inputArgs(src2), '-ss', String(start), '-i', src2.input, '-t', String(len),++ '-map', '0:v?', '-map', '0:a?', '-c', 'copy', '-avoid_negative_ts', 'make_zero', ...(ext === '.mp4' ? ['-movflags', '+faststart'] : []), out],++ { signal: ctx.signal, timeoutMs: budgetMs((Number(src.size_bytes) || 0) / parts * 2) });++ if (ctx.signal.aborted) throw ctx.signal.reason || new Error('aborted');++ if (!r.ok || !existing(out) || fs.statSync(out).size === 0) {++ try { fs.unlinkSync(out); } catch { /* none */ }++ throw new JobError('ffmpeg_failed', `part ${part}: ffmpeg exited ${r.code}: ${String(r.stderr).trim().split('\n').slice(-2).join(' ').slice(0, 300)}`);++ }++ await adopt({++ src, file: out, ext, job, variant: null,++ relation: { method: 'split', part, parts, start_seconds: start, duration_seconds: len },++ metadata: { part, parts, start_seconds: start, duration_seconds: len },++ saveCheckpoint: ctx.saveCheckpoint,++ checkpointFor: (id, size) => ({ parts: [...cp.parts, { part, object_id: id, start_seconds: start, duration_seconds: len, size_bytes: size }] }),++ });++ cp.parts = ctx.checkpoint.parts;++ }++ cleanupWork(job.id);++ const list = [...cp.parts].sort((a, b) => a.part - b.part);++ return {++ source_id: src.id, method: 'split', parts: list,++ oversized_parts: list.filter(p => p.size_bytes > maxBytes).map(p => p.part),++ source_unchanged: true,++ note: 'The parts are private objects. The source is unchanged: it stays as it is until its owner changes it.',++ };++}++++// ── object.remux ─────────────────────────────────────────────++++function validateRemux({ obj }) {++ if (!obj) throw new JobError('media.job.invalid', 'object.remux needs object_id', { permanent: true });++ if (!isMediaObject(obj)) throw new JobError('media.job.invalid', 'Only vod/clip objects (or video/audio objects) can be remuxed', { permanent: true });++ return {};++}++++async function runRemux(job, ctx) {++ const src = loadSource(job);++ if (ctx.checkpoint && ctx.checkpoint.object_id) { // finished before an interruption++ cleanupWork(job.id);++ return { source_id: src.id, method: 'remux', ...ctx.checkpoint, source_unchanged: true };++ }++ const source = await resolveSource(src);++ if (!source) throw new JobError('media_unavailable', 'The source bytes are unavailable (no local file and no cloud copy)', { permanent: true });++ checkRoom(src, Number(src.size_bytes) || 0);++ const ext = extFor(src, source.name);++ const out = path.join(workDir(job.id), `remux${ext}`);++ const r = await ffmpeg(['-y', '-nostdin', '-v', 'error', ...inputArgs(source), '-i', source.input, '-map', '0:v?', '-map', '0:a?', '-c', 'copy',++ '-fflags', '+genpts', ...(ext === '.mp4' ? ['-movflags', '+faststart'] : []), out], { signal: ctx.signal, timeoutMs: budgetMs(src.size_bytes) });++ if (ctx.signal.aborted) throw ctx.signal.reason || new Error('aborted');++ if (!r.ok || !existing(out) || fs.statSync(out).size === 0) {++ cleanupWork(job.id);++ throw new JobError('ffmpeg_failed', `ffmpeg exited ${r.code}: ${String(r.stderr).trim().split('\n').slice(-2).join(' ').slice(0, 300)}`);++ }++ const duration = await probeDuration(out);++ const made = await adopt({++ src, file: out, ext, job, variant: 'remux',++ relation: { method: 'remux', duration_seconds: duration },++ metadata: { duration_seconds: duration },++ saveCheckpoint: ctx.saveCheckpoint,++ checkpointFor: (id, size) => ({ object_id: id, size_bytes: size, duration_seconds: duration }),++ });++ cleanupWork(job.id);++ return { source_id: src.id, method: 'remux', object_id: made.id, size_bytes: made.size, duration_seconds: duration, source_unchanged: true };++}++++module.exports = {++ split: { lane: 'heavy', maxAttempts: 3, timeoutMs: 6 * 3600 * 1000, needsObject: true, validate: validateSplit, run: runSplit },++ remux: { lane: 'heavy', maxAttempts: 3, timeoutMs: 3 * 3600 * 1000, needsObject: true, validate: validateRemux, run: runRemux },++ resolveSource, isMediaObject, extFor,++};nnew file mode 100644@@@ -0,0 +1,111 @@++/**++ * OpenVibe.Media — job type invariant.scan: the size-invariant validator (light lane)++ *++ * Scans one tenant's public playback objects (server/objects/invariant.js: ready vods/clips that are++ * public or unlisted), records media_invariant_violations, and PROPOSES one job per object above the++ * max (MEDIA_PUBLIC_OBJECT_MAX_MB, 500 MB):++ *++ * object.split when the duration is known: stream-copy parts of about the target size++ * (MEDIA_PUBLIC_OBJECT_TARGET_MB), each a new private object derived from the source++ * object.remux when it is not: a stream-copy remux writes the index and duration, so a split can be++ * planned from the remuxed copy++ *++ * Proposals wait in status `proposed` for the owner (approve -> queued, cancel -> cancelled): nothing++ * is split, remuxed, re-encoded or made private automatically. Idempotency key++ * invariant:<object id>:<type>, so a later scan never proposes the same thing twice and an owner's++ * "no" (cancelled) stands. A proposal whose object stopped violating (made private, deleted, shrunk)++ * is withdrawn (cancelled by system:invariant.scan).++ *++ * result { thresholds, counts, violations, proposed, already_proposed, withdrawn, by_type }++ */++'use strict';++++const db = require('../db/database');++const invariant = require('../objects/invariant');++const model = require('../objects/model');++const queue = require('./queue');++++const MAX_PARTS = 1000;++const SYSTEM = 'system:invariant.scan';++++/** What to propose for one violating object: { type, params }. */++function planFor(obj, t = invariant.thresholds()) {++ const md = model.parseJson(obj.metadata, {});++ const size = Number(obj.size_bytes) || 0;++ const duration = Number(md.duration_seconds) || 0;++ const target = Math.max(1, t.targetBytes);++ const parts = Math.min(MAX_PARTS, Math.max(2, Math.ceil(size / target)));++ const base = { source_size_bytes: size, target_bytes: t.targetBytes, max_bytes: t.maxBytes, reason: 'public_object_too_large' };++ if (duration > 0) {++ return { type: 'object.split', params: { ...base, parts, segment_seconds: Math.max(1, Math.ceil(duration / parts)), source_duration_seconds: duration } };++ }++ return { type: 'object.remux', params: { ...base, note: 'duration unknown: a stream-copy remux writes it (and the seek index), then a split can be planned' } };++}++++function proposalKey(objectId, type) {++ return `invariant:${objectId}:${type}`;++}++++/**++ * The validator. `dryRun` writes nothing (no violation rows, no proposals) and returns what it would++ * propose. Used by the job below and by scripts/media-jobs.js scan.++ */++function scanTenant(appId, { dryRun = false, scanJobId = null } = {}) {++ const report = invariant.scan({ record: !dryRun, appId });++ const t = report.thresholds;++ const out = {++ thresholds: t, counts: report.counts, violations: 0, proposed: 0, already_proposed: 0, withdrawn: 0,++ by_type: {}, dry_run: dryRun, proposals: [],++ };++ const violating = new Set();++ for (const v of report.objects) {++ if (v.level !== 'violation') continue;++ out.violations++;++ violating.add(v.object_id);++ const obj = model.getObject(v.object_id);++ if (!obj) continue;++ const plan = planFor(obj, t);++ out.by_type[plan.type] = (out.by_type[plan.type] || 0) + 1;++ const key = proposalKey(obj.id, plan.type);++ if (dryRun) {++ const prior = db.get('SELECT id, status FROM media_jobs WHERE app_id = ? AND idempotency_key = ?', [appId, key]);++ if (prior) out.already_proposed++; else out.proposed++;++ out.proposals.push({ object_id: obj.id, legacy_ref: obj.legacy_ref, size_bytes: obj.size_bytes, type: plan.type, params: plan.params, existing: prior || null });++ continue;++ }++ try {++ const r = queue.enqueue({++ appId, type: plan.type, objectId: obj.id, params: plan.params, status: 'proposed',++ idempotencyKey: key, createdBy: scanJobId ? `${SYSTEM}:${scanJobId}` : SYSTEM,++ });++ if (r.created) out.proposed++; else out.already_proposed++;++ } catch (err) {++ if (err.code !== 'media.job.idempotency_conflict') throw err;++ out.already_proposed++; // proposed before with other numbers: the earlier proposal (and any decision on it) stands++ }++ }++ // Withdraw open proposals whose object no longer breaks the policy.++ const open = db.all(`SELECT id, object_id FROM media_jobs WHERE app_id = ? AND status = 'proposed' AND created_by LIKE ?`, [appId, `${SYSTEM}%`]);++ for (const p of open) {++ if (violating.has(p.object_id)) continue;++ if (dryRun) { out.withdrawn++; continue; }++ const r = queue.cancel(p.id, { by: SYSTEM, reason: 'the object no longer breaks the public size policy' });++ if (r && r.job && r.job.status === 'cancelled') out.withdrawn++;++ }++ if (!dryRun) delete out.proposals;++ return out;++}++++async function run(job) {++ return scanTenant(job.app_id, { scanJobId: job.id });++}++++function validate({ obj }) {++ if (obj) throw new queue.JobError('media.job.invalid', 'invariant.scan covers the whole tenant: leave object_id out', { permanent: true });++ return {};++}++++module.exports = {++ spec: { lane: 'light', maxAttempts: 2, timeoutMs: 10 * 60 * 1000, needsObject: false, validate, run },++ planFor, scanTenant, proposalKey,++};nnew file mode 100644@@@ -0,0 +1,372 @@++/**++ * OpenVibe.Media — job queue (media_jobs; docs/object-model.md#jobs)++ *++ * States:++ * proposed waiting for its owner to decide (approve -> queued, cancel -> cancelled). The worker never++ * picks it up: the size-invariant validator proposes split/remux jobs this way.++ * queued due when run_after has passed++ * running held by the worker (lease_until renewed while it runs)++ * succeeded | failed | cancelled finished++ *++ * Every state change and its media.job.<transition> event commit in ONE SQLite transaction (the same++ * rule as webhooks.announce(), see events.js): an event exists if and only if its change committed.++ * The relay is woken after the commit. Job events go to OpenVibe.Events only (no app webhook).++ *++ * Idempotency: a key is unique per tenant. A repeat with the same key and the same request (type,++ * object, params) answers with the job it made; a different request under a used key is a conflict.++ * `dedupeActive` instead joins an identical job that is still queued or running (thumbnail requests).++ */++'use strict';++++const crypto = require('crypto');++const { EventEmitter } = require('events');++const { ids } = require('openvibe-contracts');++const db = require('../db/database');++const events = require('../events');++++const STATUSES = ['proposed', 'queued', 'running', 'succeeded', 'failed', 'cancelled'];++const FINISHED = ['succeeded', 'failed', 'cancelled'];++const ACTIVE = ['queued', 'running'];++const MAX_ACTIVE_PER_TENANT = 50; // queued + running jobs one tenant may have (proposals excluded)++const RESULT_EVENT_MAX = 8192; // a larger result is left out of the event payload (GET the job for it)++++const bus = new EventEmitter();++bus.setMaxListeners(0);++++class JobError extends Error {++ /** code: stable machine code; permanent: never retried; status: HTTP status for API answers. */++ constructor(code, message, { permanent = false, status = 400, retryAfterS = null } = {}) {++ super(message);++ this.code = code;++ this.permanent = permanent;++ this.status = status;++ this.retryAfterS = retryAfterS;++ }++}++++// ── Types (registered by server/jobs/types.js) ───────────────++++const TYPES = new Map();++let builtinsLoaded = false;++function loadBuiltins() {++ if (builtinsLoaded) return;++ builtinsLoaded = true;++ require('./types');++}++++/**++ * spec: { lane: 'light'|'heavy', maxAttempts, timeoutMs, needsObject, validate(ctx) -> params, run(job, ctx) -> result,++ * backoffS(attempt) }++ */++function register(type, spec) {++ TYPES.set(type, { lane: 'light', maxAttempts: 3, timeoutMs: 10 * 60 * 1000, needsObject: false, ...spec, type });++}++function typeSpec(type) {++ loadBuiltins();++ return TYPES.get(type) || null;++}++function typeNames() {++ loadBuiltins();++ return [...TYPES.keys()];++}++++// ── Helpers ──────────────────────────────────────────────────++++function parseJson(s, fallback) {++ if (s == null || s === '') return fallback;++ try { return JSON.parse(s); } catch { return fallback; }++}++++/** JSON with sorted keys, so the same params always hash the same. */++function canonical(v) {++ if (Array.isArray(v)) return `[${v.map(canonical).join(',')}]`;++ if (v && typeof v === 'object') return `{${Object.keys(v).sort().map(k => `${JSON.stringify(k)}:${canonical(v[k])}`).join(',')}}`;++ return JSON.stringify(v === undefined ? null : v);++}++function requestHash(type, objectId, params) {++ return crypto.createHash('sha256').update(canonical({ type, object_id: objectId || null, params: params || {} })).digest('hex');++}++++function get(id) {++ return id ? db.get('SELECT * FROM media_jobs WHERE id = ?', [String(id)]) : null;++}++function getForApp(id, appId) {++ const row = get(id);++ return row && row.app_id === appId ? row : null;++}++++function jobPublic(row) {++ if (!row) return null;++ return {++ id: row.id,++ app_id: row.app_id,++ object_id: row.object_id || null,++ type: row.job_type,++ status: row.status,++ params: parseJson(row.params, {}),++ result: parseJson(row.result, null),++ error: row.error || null,++ error_code: row.error_code || null,++ attempts: row.attempts,++ max_attempts: row.max_attempts,++ run_after: row.run_after || null,++ cancel_requested: !!row.cancel_requested,++ idempotency_key: row.idempotency_key || null,++ created_by: row.created_by || null,++ owner_user_id: row.owner_user_id ?? null,++ decided_by: row.decided_by || null,++ decided_at: row.decided_at || null,++ created_at: row.created_at,++ updated_at: row.updated_at,++ started_at: row.started_at || null,++ finished_at: row.finished_at || null,++ };++}++++function eventPayload(row) {++ const pub = jobPublic(row);++ if (pub.result && JSON.stringify(pub.result).length > RESULT_EVENT_MAX) pub.result = { omitted: true, reason: 'larger than the event limit; GET the job' };++ return pub;++}++++/**++ * Run `change()` and, when it reports a change, queue media.job.<transition> for the job, in one++ * transaction; wake the relay and tell waiters after the commit. Returns the job row, or null when++ * `change()` changed nothing (a lost race: the caller decides what that means).++ */++function commit(id, transition, change) {++ let row = null;++ db.getDb().transaction(() => {++ if (!change()) return;++ row = get(id);++ if (transition) events.recordJob(transition, eventPayload(row));++ })();++ if (row) {++ if (transition) events.kick();++ bus.emit('change', row);++ if (FINISHED.includes(row.status)) bus.emit(`finished:${row.id}`, row);++ }++ return row;++}++++function update(id, fromStatuses, sets, params = []) {++ const from = Array.isArray(fromStatuses) ? fromStatuses : [fromStatuses];++ return db.run(`UPDATE media_jobs SET ${sets}, updated_at = CURRENT_TIMESTAMP WHERE id = ? AND status IN (${from.map(() => '?').join(', ')})`,++ [...params, id, ...from]).changes > 0;++}++++// ── Enqueue ──────────────────────────────────────────────────++++/**++ * enqueue({ appId, type, objectId, params, status: 'queued'|'proposed', idempotencyKey, dedupeActive,++ * createdBy, ownerUserId, maxAttempts, runAfterS })++ * -> { job, created, replayed, deduped }++ * Throws JobError: media.job.unknown_type, media.job.idempotency_conflict (409), media.job.too_many (429).++ * `params` must already be validated (routes.js / the validator do that through the type's validate()).++ */++function enqueue({ appId, type, objectId = null, params = {}, status = 'queued', idempotencyKey = null, dedupeActive = false,++ createdBy = null, ownerUserId = null, maxAttempts = null, runAfterS = 0 }) {++ const spec = typeSpec(type);++ if (!spec) throw new JobError('media.job.unknown_type', `Unknown job type ${type}`, { status: 400, permanent: true });++ if (!['queued', 'proposed'].includes(status)) throw new Error(`a job starts queued or proposed, not ${status}`);++ const key = idempotencyKey == null || idempotencyKey === '' ? null : String(idempotencyKey).slice(0, 200);++ const hash = requestHash(type, objectId, params);++ let out = null;++ const id = `mjob_${ids.ulid()}`;++ db.getDb().transaction(() => {++ if (key) {++ const prior = db.get('SELECT * FROM media_jobs WHERE app_id = ? AND idempotency_key = ?', [appId, key]);++ if (prior) {++ if (prior.request_hash !== hash) {++ throw new JobError('media.job.idempotency_conflict', 'This Idempotency-Key was used for a different job request', { status: 409, permanent: true });++ }++ out = { job: prior, created: false, replayed: true, deduped: false };++ return;++ }++ }++ if (dedupeActive) {++ const same = db.get(`SELECT * FROM media_jobs WHERE app_id = ? AND job_type = ? AND request_hash = ? AND status IN ('queued', 'running')++ ORDER BY id DESC LIMIT 1`, [appId, type, hash]);++ if (same) { out = { job: same, created: false, replayed: false, deduped: true }; return; }++ }++ if (status === 'queued') {++ const active = db.get("SELECT COUNT(*) AS n FROM media_jobs WHERE app_id = ? AND status IN ('queued', 'running')", [appId]).n;++ if (active >= MAX_ACTIVE_PER_TENANT) {++ throw new JobError('media.job.too_many', `At most ${MAX_ACTIVE_PER_TENANT} jobs may be queued or running per tenant`, { status: 429, retryAfterS: 60 });++ }++ }++ db.run(`INSERT INTO media_jobs (id, app_id, object_id, job_type, status, idempotency_key, request_hash, params, max_attempts,++ run_after, created_by, owner_user_id)++ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ${runAfterS > 0 ? `datetime('now', '+${Math.round(runAfterS)} seconds')` : 'NULL'}, ?, ?)`,++ [id, appId, objectId, type, status, key, hash, JSON.stringify(params || {}),++ Math.max(1, Math.min(10, Number(maxAttempts) || spec.maxAttempts)), createdBy, ownerUserId ?? null]);++ const row = get(id);++ events.recordJob(status, eventPayload(row));++ out = { job: row, created: true, replayed: false, deduped: false };++ })();++ if (out.created) { events.kick(); bus.emit('change', out.job); }++ return out;++}++++// ── Owner decisions ──────────────────────────────────────────++++/** proposed -> queued. Returns the job, or null when it is not a proposal (any more). */++function approve(id, { by = null } = {}) {++ return commit(id, 'queued', () => update(id, 'proposed', "status = 'queued', decided_by = ?, decided_at = CURRENT_TIMESTAMP, run_after = NULL", [by]));++}++++/**++ * Cancel by the owner. proposed/queued -> cancelled at once. A running job gets cancel_requested and++ * stops at its next check (the worker also aborts it in-process); { job, pending: true } until then.++ * Finished jobs: { job, finished: true } and nothing changes.++ */++function cancel(id, { by = null, reason = null } = {}) {++ const row = get(id);++ if (!row) return null;++ if (FINISHED.includes(row.status)) return { job: row, finished: true };++ if (row.status === 'running') {++ db.run('UPDATE media_jobs SET cancel_requested = 1, decided_by = COALESCE(decided_by, ?), updated_at = CURRENT_TIMESTAMP WHERE id = ?', [by, id]);++ try { require('./worker').abort(id); } catch { /* the heartbeat sees the flag */ }++ return { job: get(id), pending: true };++ }++ const done = commit(id, 'cancelled', () => update(id, ['proposed', 'queued'],++ `status = 'cancelled', finished_at = CURRENT_TIMESTAMP, decided_by = ?, decided_at = CURRENT_TIMESTAMP,++ error = COALESCE(?, error), error_code = CASE WHEN ? IS NULL THEN error_code ELSE 'cancelled' END`, [by, reason, reason]));++ if (done) return { job: done };++ return cancel(id, { by, reason }); // it moved on meanwhile (picked up): try again from its new state++}++++// ── Worker side ──────────────────────────────────────────────++++/**++ * Take the oldest due queued job of these types (or exactly `id`): running, attempts + 1, leased.++ * Returns the row or null.++ */++function claim(types, { leaseS = 120, id = null } = {}) {++ if (!id && !types.length) return null;++ let row = null;++ db.getDb().transaction(() => {++ const next = id++ ? db.get("SELECT id FROM media_jobs WHERE id = ? AND status = 'queued'", [id])++ : db.get(`SELECT id FROM media_jobs WHERE status = 'queued' AND job_type IN (${types.map(() => '?').join(', ')})++ AND (run_after IS NULL OR run_after <= datetime('now')) ORDER BY id LIMIT 1`, types);++ if (!next) return;++ if (!update(next.id, 'queued', `status = 'running', attempts = attempts + 1, started_at = COALESCE(started_at, CURRENT_TIMESTAMP),++ lease_until = datetime('now', '+${Math.round(leaseS)} seconds'), run_after = NULL`)) return;++ row = get(next.id);++ events.recordJob('started', eventPayload(row));++ })();++ if (row) { events.kick(); bus.emit('change', row); }++ return row;++}++++/** Extend a running job's lease. Returns { cancelRequested } (false when it is no longer running). */++function renew(id, { leaseS = 120 } = {}) {++ db.run(`UPDATE media_jobs SET lease_until = datetime('now', '+${Math.round(leaseS)} seconds') WHERE id = ? AND status = 'running'`, [id]);++ const row = get(id);++ return { running: !!row && row.status === 'running', cancelRequested: !!(row && row.cancel_requested) };++}++++/** Save handler progress; `alsoInTx()` (optional) runs in the same transaction (e.g. the object a part became). */++function saveCheckpoint(id, checkpoint, alsoInTx = null) {++ db.getDb().transaction(() => {++ if (alsoInTx) alsoInTx();++ db.run('UPDATE media_jobs SET checkpoint = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ?', [JSON.stringify(checkpoint), id]);++ })();++}++++function succeed(id, result) {++ return commit(id, 'succeeded', () => update(id, 'running',++ "status = 'succeeded', result = ?, error = NULL, error_code = NULL, lease_until = NULL, finished_at = CURRENT_TIMESTAMP",++ [JSON.stringify(result ?? null)]));++}++++/** A failed attempt: queued again after `retryInS` (media.job.retrying), or failed for good. */++function fail(id, { message, code = null, retryInS = null, result = null }) {++ const msg = String(message || 'failed').slice(0, 1000);++ if (retryInS != null) {++ return commit(id, 'retrying', () => update(id, 'running',++ `status = 'queued', error = ?, error_code = ?, lease_until = NULL, run_after = datetime('now', '+${Math.max(0, Math.round(retryInS))} seconds')`,++ [msg, code]));++ }++ return commit(id, 'failed', () => update(id, 'running',++ "status = 'failed', error = ?, error_code = ?, result = COALESCE(?, result), lease_until = NULL, finished_at = CURRENT_TIMESTAMP",++ [msg, code, result == null ? null : JSON.stringify(result)]));++}++++/** A running job that stopped because its owner cancelled it. */++function markCancelled(id, { result = null } = {}) {++ return commit(id, 'cancelled', () => update(id, 'running',++ "status = 'cancelled', error = 'cancelled by its owner', error_code = 'cancelled', result = COALESCE(?, result), lease_until = NULL, finished_at = CURRENT_TIMESTAMP",++ [result == null ? null : JSON.stringify(result)]));++}++++/**++ * Running jobs nobody holds any more: every one at boot (`all`), otherwise those whose lease ran out.++ * `except` = ids this process is running. Each is retried if it has attempts left, else failed.++ */++function recoverInterrupted({ all = false, except = new Set(), retryInS = 30 } = {}) {++ const rows = db.all(`SELECT * FROM media_jobs WHERE status = 'running'${all ? '' : " AND (lease_until IS NULL OR lease_until < datetime('now'))"}`);++ let n = 0;++ for (const row of rows) {++ if (except.has(row.id)) continue;++ const why = all ? 'interrupted by a restart' : 'the worker stopped renewing its lease';++ if (row.cancel_requested) markCancelled(row.id);++ else if (row.attempts < row.max_attempts) fail(row.id, { message: why, code: 'interrupted', retryInS });++ else fail(row.id, { message: why, code: 'interrupted' });++ n++;++ }++ return n;++}++++/** Wait until the job is finished (or `timeoutMs` passes). Resolves with the row as it is then. */++function waitFor(id, timeoutMs = 60000) {++ const now = get(id);++ if (!now || FINISHED.includes(now.status)) return Promise.resolve(now);++ return new Promise((resolve) => {++ const ev = `finished:${id}`;++ const done = (row) => { clearTimeout(t); bus.removeListener(ev, done); resolve(row || get(id)); };++ const t = setTimeout(() => done(null), timeoutMs);++ if (t.unref) t.unref();++ bus.on(ev, done);++ });++}++++// ── Reads ────────────────────────────────────────────────────++++/** Cursor list, newest first. filters: status, type, objectId, ownerUserId (acting user: own jobs + jobs on own objects). */++function list(appId, { status, type, objectId, cursor, limit = 50, actingUserId = null } = {}) {++ const conds = ['j.app_id = ?'], params = [appId];++ if (cursor) { conds.push('j.id < ?'); params.push(String(cursor)); }++ if (status) { conds.push('j.status = ?'); params.push(String(status)); }++ if (type) { conds.push('j.job_type = ?'); params.push(String(type)); }++ if (objectId) { conds.push('j.object_id = ?'); params.push(String(objectId)); }++ if (actingUserId != null) {++ conds.push('(j.owner_user_id = ? OR EXISTS (SELECT 1 FROM media_objects o WHERE o.id = j.object_id AND o.owner_user_id = ?))');++ params.push(actingUserId, actingUserId);++ }++ const n = Math.min(Math.max(parseInt(limit, 10) || 50, 1), 200);++ const rows = db.all(`SELECT j.* FROM media_jobs j WHERE ${conds.join(' AND ')} ORDER BY j.id DESC LIMIT ?`, [...params, n + 1]);++ const page = rows.slice(0, n);++ return { jobs: page, next_cursor: rows.length > n ? page[page.length - 1].id : null, limit: n };++}++++function counts() {++ const out = Object.fromEntries(STATUSES.map(s => [s, 0]));++ for (const r of db.all('SELECT status, COUNT(*) AS n FROM media_jobs GROUP BY status')) out[r.status] = r.n;++ return out;++}++++/** Delete finished jobs of high-volume types older than `days` (thumbnail requests). Others are kept. */++function prune({ days = 30, types = ['thumbnail.regenerate'] } = {}) {++ return db.run(`DELETE FROM media_jobs WHERE status IN ('succeeded', 'failed', 'cancelled') AND job_type IN (${types.map(() => '?').join(', ')})++ AND finished_at < datetime('now', ?)`, [...types, `-${Math.max(1, days)} days`]).changes;++}++++module.exports = {++ STATUSES, FINISHED, ACTIVE, MAX_ACTIVE_PER_TENANT, JobError, bus,++ register, typeSpec, typeNames, requestHash, parseJson,++ get, getForApp, jobPublic, list, counts,++ enqueue, approve, cancel,++ claim, renew, saveCheckpoint, succeed, fail, markCancelled, recoverInterrupted, waitFor, prune,++};nnew file mode 100644@@@ -0,0 +1,137 @@++/**++ * OpenVibe.Media — Jobs API v2 (mounted at /api/v2/:app/jobs; docs/object-model.md#jobs)++ *++ * GET / cursor list, newest first (?status&type&object_id&limit&cursor)++ * POST / { type, object_id?, params?, max_attempts? } + Idempotency-Key++ * -> 202 { job } new | 200 { job } + Idempotent-Replayed: true++ * GET /:jobId one job++ * POST /:jobId/approve a proposal (status proposed) -> queued++ * POST /:jobId/cancel cancel (also DELETE /:jobId): proposed/queued at once -> 200;++ * running -> 202 { job } with cancel_requested until it stops; finished -> 409++ *++ * Auth as the objects API: the app key or a Network service token granting media.object.read (reads)++ * or media.object.upload (create, approve, cancel) for namespace :app; a developer project's app++ * tokens on its own tenant. With X-OV-User-Id (app key) the caller acts for one of the app's users:++ * it sees and decides only jobs it created or jobs on objects it owns, and creates jobs on its own++ * objects only. Errors are RFC 9457 problems.++ */++'use strict';++++const express = require('express');++const { http } = require('openvibe-contracts');++const model = require('../objects/model');++const queue = require('./queue');++const { tenantAuth, tenantCors } = require('../auth');++++const write = tenantAuth({ capability: 'media.object.upload' });++const read = tenantAuth({ capability: 'media.object.read' });++++function problem(res, status, code, detail, extra) {++ return http.sendProblem(res, status, code, { detail, extra });++}++++function who(req) {++ if (req.principal) return req.principal.sub;++ return `app:${req.appId}${req.authType === 'user' ? `:user:${req.userId}` : ''}`;++}++++function ownsJob(req, job) {++ if (req.authType !== 'user') return true;++ if (job.owner_user_id != null && job.owner_user_id === req.userId) return true;++ const obj = job.object_id ? model.getObject(job.object_id) : null;++ return !!(obj && obj.owner_user_id != null && obj.owner_user_id === req.userId);++}++++function load(req, res) {++ const job = queue.getForApp(String(req.params.jobId || ''), req.appId);++ if (!job || !ownsJob(req, job)) { problem(res, 404, 'media.job.not_found', 'No such job in this namespace'); return null; }++ return job;++}++++function sendJobError(res, err) {++ if (err instanceof queue.JobError) {++ if (err.retryAfterS) res.set('Retry-After', String(err.retryAfterS));++ return problem(res, err.status || 400, err.code, err.message);++ }++ console.error('[Jobs] API error:', err.message);++ return problem(res, 500, 'media.job.failed_request', 'Job request failed');++}++++const router = express.Router({ mergeParams: true });++router.use(tenantCors);++++router.get('/', read, (req, res) => {++ try {++ const q = req.query;++ if (q.cursor && !/^mjob_[0-9A-Za-z_]{1,40}$/.test(String(q.cursor))) return problem(res, 400, 'media.job.invalid', 'Bad cursor');++ if (q.status && !queue.STATUSES.includes(String(q.status))) return problem(res, 400, 'media.job.invalid', `status must be one of ${queue.STATUSES.join(', ')}`);++ const out = queue.list(req.appId, {++ status: q.status, type: q.type, objectId: q.object_id, cursor: q.cursor, limit: q.limit,++ actingUserId: req.authType === 'user' ? req.userId : null,++ });++ res.json({ jobs: out.jobs.map(queue.jobPublic), next_cursor: out.next_cursor, limit: out.limit });++ } catch (err) { sendJobError(res, err); }++});++++router.post('/', write, (req, res) => {++ try {++ const b = req.body || {};++ const type = String(b.type || '');++ const spec = queue.typeSpec(type);++ if (!spec) return problem(res, 400, 'media.job.unknown_type', `type must be one of ${queue.typeNames().join(', ')}`);++ let obj = null;++ if (b.object_id != null && b.object_id !== '') {++ obj = model.resolveObject(String(b.object_id), req.appId);++ if (!obj || (req.authType === 'user' && obj.visibility === 'private' && obj.owner_user_id !== req.userId)) {++ return problem(res, 404, 'media.object.not_found', 'No such object in this namespace');++ }++ if (req.authType === 'user' && obj.owner_user_id !== req.userId) return problem(res, 403, 'media.object.forbidden', 'Not your object');++ } else if (spec.needsObject) {++ return problem(res, 400, 'media.job.invalid', `${type} needs object_id`);++ }++ if (b.params != null && (typeof b.params !== 'object' || Array.isArray(b.params) || JSON.stringify(b.params).length > 16384)) {++ return problem(res, 400, 'media.job.invalid', 'params must be an object of at most 16 KB');++ }++ const params = spec.validate ? spec.validate({ appId: req.appId, obj, params: b.params || {} }) : (b.params || {});++ const key = req.headers['idempotency-key'] || b.idempotency_key || null;++ if (key != null && !/^[\x21-\x7e]{1,200}$/.test(String(key))) return problem(res, 400, 'media.job.invalid', 'Idempotency-Key must be 1-200 visible ASCII characters');++ const r = queue.enqueue({++ appId: req.appId, type, objectId: obj ? obj.id : null, params, idempotencyKey: key,++ createdBy: who(req), ownerUserId: req.authType === 'user' ? req.userId : null, maxAttempts: b.max_attempts,++ });++ if (r.replayed) res.set('Idempotent-Replayed', 'true');++ else require('./worker').kick();++ res.status(r.created ? 202 : 200).json({ job: queue.jobPublic(r.job) });++ } catch (err) { sendJobError(res, err); }++});++++router.get('/:jobId', read, (req, res) => {++ const job = load(req, res);++ if (job) res.json({ job: queue.jobPublic(job) });++});++++router.post('/:jobId/approve', write, (req, res) => {++ try {++ const job = load(req, res);++ if (!job) return;++ if (job.status !== 'proposed') return problem(res, 409, 'media.job.not_proposed', `The job is ${job.status}; only proposals are approved`);++ const out = queue.approve(job.id, { by: who(req) });++ if (!out) return problem(res, 409, 'media.job.not_proposed', 'The job was decided meanwhile');++ require('./worker').kick();++ res.json({ job: queue.jobPublic(out) });++ } catch (err) { sendJobError(res, err); }++});++++function cancelHandler(req, res) {++ try {++ const job = load(req, res);++ if (!job) return;++ const out = queue.cancel(job.id, { by: who(req) });++ if (out.finished) return problem(res, 409, 'media.job.finished', `The job already ${out.job.status}`);++ res.status(out.pending ? 202 : 200).json({ job: queue.jobPublic(out.job) });++ } catch (err) { sendJobError(res, err); }++}++router.post('/:jobId/cancel', write, cancelHandler);++router.delete('/:jobId', write, cancelHandler);++++module.exports = router;nnew file mode 100644@@@ -0,0 +1,62 @@++/**++ * OpenVibe.Media — job type thumbnail.regenerate (light lane)++ *++ * The thumbnail (re)generation POST /api/v1/:app/thumbnails/:kind/:id did inline now runs as a job:++ * the route enqueues it (joining an identical job that is still queued or running) and waits for it,++ * so its answer is unchanged. The v2 jobs API can queue it too, for a vod or clip object.++ *++ * params { kind: 'vod' | 'clip', id: <row id> } (derived from the object's legacy ref when omitted)++ * result { url, kind, id }++ * Failures: media_unavailable (no local file and no cloud copy; permanent), not_found (permanent),++ * generate_failed (ffmpeg produced no frame; retried).++ */++'use strict';++++const db = require('../db/database');++const { JobError } = require('./queue');++++function targetOf(obj, params = {}) {++ let kind = params.kind != null ? String(params.kind) : null;++ let id = params.id != null ? Number(params.id) : null;++ if ((!kind || id == null) && obj) {++ const ref = require('../objects/model').parseLegacyRef(obj.legacy_ref);++ if (ref && (ref.kind === 'vod' || ref.kind === 'clip')) { kind = kind || ref.kind; id = id != null ? id : Number(ref.id); }++ }++ if (kind !== 'vod' && kind !== 'clip') throw new JobError('media.job.invalid', "thumbnail.regenerate needs a vod or clip (params.kind 'vod' | 'clip' and params.id, or a vod/clip object)", { permanent: true });++ if (!Number.isInteger(id) || id < 1) throw new JobError('media.job.invalid', 'thumbnail.regenerate needs params.id (the vod or clip id)', { permanent: true });++ return { kind, id };++}++++function rowOf(appId, kind, id) {++ return kind === 'vod' ? db.getVodById(id, appId) : db.getClipById(id, appId);++}++++/** Normalized params, or JobError (400). The row must exist in this tenant (and match the object when one is named). */++function validate({ appId, obj, params }) {++ const t = targetOf(obj, params || {});++ const row = rowOf(appId, t.kind, t.id);++ if (!row) throw new JobError('media.job.not_found', `${t.kind} ${t.id} not found`, { status: 404, permanent: true });++ if (obj && row.object_id && row.object_id !== obj.id) throw new JobError('media.job.invalid', `the object is not ${t.kind} ${t.id}`, { permanent: true });++ return t;++}++++async function run(job) {++ const { kind, id } = targetOf(null, job.params);++ const row = rowOf(job.app_id, kind, id);++ if (!row) throw new JobError('not_found', `${kind} ${id} not found`, { permanent: true });++ const thumbService = require('../thumbnails/thumbnail-service');++ let source = null;++ if (kind === 'clip' && row.file_path && require('fs').existsSync(row.file_path)) source = { value: row.file_path };++ else source = await require('../vod/vod-storage').resolveMediaSource(row);++ if (!source) throw new JobError('media_unavailable', 'Media file unavailable', { permanent: true, status: 404 });++ const url = kind === 'vod'++ ? await thumbService.generateVodThumbnail(id, source.value)++ : await thumbService.generateClipThumbnail(id, source.value);++ if (!url) throw new JobError('generate_failed', 'Failed to generate thumbnail');++ return { url, kind, id };++}++++module.exports = {++ spec: { lane: 'light', maxAttempts: 2, timeoutMs: 2 * 60 * 1000, needsObject: false, backoffS: () => 20, validate, run },++ targetOf,++};nnew file mode 100644@@@ -0,0 +1,18 @@++/**++ * OpenVibe.Media — the job types the worker runs (docs/object-model.md#jobs).++ *++ * thumbnail.regenerate light the thumbnail (re)generation the v1 thumbnail route used to do inline++ * invariant.scan light the size-invariant validator: proposes object.split / object.remux++ * object.split heavy stream-copy parts of a large vod/clip as new private objects++ * object.remux heavy a stream-copy remux of a vod/clip as a new private object++ */++'use strict';++++const queue = require('./queue');++++queue.register('thumbnail.regenerate', require('./thumbnail').spec);++queue.register('invariant.scan', require('./invariant-scan').spec);++queue.register('object.split', require('./derive').split);++queue.register('object.remux', require('./derive').remux);++++module.exports = { names: () => queue.typeNames() };nnew file mode 100644@@@ -0,0 +1,202 @@++/**++ * OpenVibe.Media — job worker (docs/object-model.md#jobs)++ *++ * Runs media_jobs in-process, in two lanes so a long split never holds up a thumbnail:++ * light thumbnail.regenerate, invariant.scan MEDIA_JOBS_LIGHT_CONCURRENCY (2)++ * heavy object.split, object.remux MEDIA_JOBS_HEAVY_CONCURRENCY (1); waits while a++ * recording runs unless MEDIA_JOBS_HEAVY_WHILE_RECORDING=1++ *++ * A running job holds a lease (MEDIA_JOBS_LEASE_S) that a heartbeat renews; the heartbeat also notices++ * an owner's cancel request. At start every job left `running` by the previous process is retried (or++ * failed when out of attempts). A failed attempt is retried with backoff unless the handler says the++ * failure is permanent. Handlers get { signal, checkpoint, saveCheckpoint(cp, alsoInTx) } and resume++ * from the checkpoint on a retry.++ *++ * Scheduling: the size-invariant validator (invariant.scan) is enqueued per tenant every++ * MEDIA_INVARIANT_SCAN_HOURS (it proposes split/remux jobs; it never runs them).++ */++'use strict';++++const config = require('../config');++const db = require('../db/database');++const queue = require('./queue');++++const cfg = () => config.jobs;++const running = new Map(); // job id -> { ac, lane }++let timer = null;++let started = false;++let ticking = false;++let lastScheduleAt = 0;++let lastPruneAt = 0;++let kicked = false;++++function lanes() {++ const types = queue.typeNames();++ const by = (lane) => types.filter(t => queue.typeSpec(t).lane === lane);++ return [++ { name: 'light', types: by('light'), max: cfg().lightConcurrency },++ { name: 'heavy', types: by('heavy'), max: cfg().heavyConcurrency },++ ];++}++++function inLane(name) {++ let n = 0;++ for (const r of running.values()) if (r.lane === name) n++;++ return n;++}++++function recordingActive() {++ try { return require('../vod/recorder').activeCount() > 0; } catch { return false; }++}++++function backoffS(spec, attempt) {++ if (typeof spec.backoffS === 'function') return spec.backoffS(attempt);++ return Math.min(3600, 30 * 4 ** Math.max(0, attempt - 1)); // 30 s, 2 min, 8 min, 32 min, 1 h++}++++/** Run one claimed job to its end. Never throws. */++async function execute(row, laneName) {++ const spec = queue.typeSpec(row.job_type);++ const ac = new AbortController();++ running.set(row.id, { ac, lane: laneName });++ const leaseS = cfg().leaseS;++ let cancelRequested = !!row.cancel_requested;++ const beat = setInterval(() => {++ try {++ const r = queue.renew(row.id, { leaseS });++ if (r.cancelRequested && !cancelRequested) { cancelRequested = true; ac.abort(new Error('cancelled by its owner')); }++ } catch { /* next beat */ }++ }, Math.max(1000, Math.floor(leaseS * 1000 / 3)));++ if (beat.unref) beat.unref();++ const timeoutMs = spec ? spec.timeoutMs : 1000;++ const limit = setTimeout(() => ac.abort(new Error(`timed out after ${Math.round(timeoutMs / 1000)} s`)), timeoutMs);++ if (limit.unref) limit.unref();++ if (cancelRequested) ac.abort(new Error('cancelled by its owner'));++++ const ctx = {++ signal: ac.signal,++ checkpoint: queue.parseJson(row.checkpoint, null),++ attempt: row.attempts,++ saveCheckpoint: (cp, alsoInTx) => { ctx.checkpoint = cp; queue.saveCheckpoint(row.id, cp, alsoInTx); },++ cancelled: () => cancelRequested,++ };++ try {++ if (!spec) throw new queue.JobError('media.job.unknown_type', `No handler for ${row.job_type}`, { permanent: true });++ const result = await spec.run(queue.jobPublic(row), ctx);++ queue.succeed(row.id, result);++ } catch (err) {++ const fresh = queue.get(row.id);++ if (cancelRequested || (fresh && fresh.cancel_requested)) {++ queue.markCancelled(row.id, { result: ctx.checkpoint ? { checkpoint: ctx.checkpoint } : null });++ } else {++ const permanent = !!(err && err.permanent);++ const attempts = fresh ? fresh.attempts : row.attempts;++ const more = !permanent && attempts < (fresh ? fresh.max_attempts : row.max_attempts);++ const retryIn = err && err.retryAfterS != null ? err.retryAfterS : backoffS(spec || {}, attempts);++ queue.fail(row.id, { message: (err && err.message) || String(err), code: (err && err.code) || null, retryInS: more ? retryIn : null });++ if (!more) console.warn(`[Jobs] ${row.job_type} ${row.id} failed: ${(err && err.message) || err}`);++ }++ } finally {++ clearInterval(beat);++ clearTimeout(limit);++ running.delete(row.id);++ kick();++ }++}++++/**++ * Run this queued job now, outside the lane limits: the v1 thumbnail route does this so a request++ * is answered as promptly as before the job system (ffmpeg per request, deduplicated per item).++ * Resolves when it finishes; resolves null when someone else already took it.++ */++async function runNow(id) {++ const row = queue.claim([], { id, leaseS: cfg().leaseS });++ if (!row) return null;++ await execute(row, 'direct');++ return queue.get(id);++}++++function scheduleInvariantScans(nowMs = Date.now()) {++ const hours = cfg().invariantScanHours;++ if (!hours || hours <= 0) return 0;++ const periodMs = hours * 3600 * 1000;++ const period = Math.floor(nowMs / periodMs);++ let n = 0;++ const tenants = db.all(`SELECT DISTINCT app_id FROM media_objects WHERE kind IN ('vod', 'clip') AND visibility != 'private' AND lifecycle_status = 'ready'`);++ for (const { app_id: appId } of tenants) {++ try {++ const r = queue.enqueue({ appId, type: 'invariant.scan', params: {}, idempotencyKey: `invariant.scan:${hours}h:${period}`, createdBy: 'system:schedule' });++ if (r.created) n++;++ } catch (err) { console.warn(`[Jobs] could not schedule invariant.scan for ${appId}: ${err.message}`); }++ }++ return n;++}++++async function tick() {++ if (ticking) { kicked = true; return; }++ ticking = true;++ try {++ do {++ kicked = false;++ queue.recoverInterrupted({ except: new Set(running.keys()) });++ const now = Date.now();++ if (now - lastScheduleAt > 10 * 60 * 1000) { lastScheduleAt = now; scheduleInvariantScans(now); }++ if (now - lastPruneAt > 6 * 3600 * 1000) {++ lastPruneAt = now;++ try { const n = queue.prune({ days: cfg().retentionDays }); if (n) console.log(`[Jobs] pruned ${n} finished thumbnail job(s)`); } catch { /* next time */ }++ }++ for (const lane of lanes()) {++ if (lane.name === 'heavy' && !cfg().heavyWhileRecording && recordingActive()) continue;++ while (inLane(lane.name) < lane.max) {++ const row = queue.claim(lane.types, { leaseS: cfg().leaseS });++ if (!row) break;++ execute(row, lane.name);++ }++ }++ } while (kicked);++ } catch (err) {++ console.warn('[Jobs] tick:', err.message);++ } finally {++ ticking = false;++ }++}++++/** Look for work now (after an enqueue or approval). */++function kick() {++ if (!started) return;++ setImmediate(() => { tick().catch(() => {}); });++}++++/** Ask a job running in this process to stop (its owner cancelled it). */++function abort(id) {++ const r = running.get(id);++ if (r) r.ac.abort(new Error('cancelled by its owner'));++ return !!r;++}++++function start() {++ if (started || !cfg().enabled) return false;++ started = true;++ const n = queue.recoverInterrupted({ all: true });++ if (n) console.log(`[Jobs] ${n} job(s) interrupted by the restart were requeued or failed`);++ lastScheduleAt = Date.now(); // the first scheduled scan waits a few minutes after boot++ timer = setInterval(() => { tick().catch(() => {}); }, cfg().pollMs);++ if (timer.unref) timer.unref();++ setTimeout(() => { tick().catch(() => {}); }, 1000).unref?.();++ console.log(`[Jobs] worker started (light ${cfg().lightConcurrency}, heavy ${cfg().heavyConcurrency}, poll ${cfg().pollMs} ms)`);++ return true;++}++++/** Stop polling and abort running jobs; they are requeued at the next start. */++function stop() {++ started = false;++ if (timer) clearInterval(timer);++ timer = null;++ for (const r of running.values()) r.ac.abort(new Error('the service is stopping'));++}++++function status() {++ return { enabled: cfg().enabled, started, running: [...running.entries()].map(([id, r]) => ({ id, lane: r.lane })), counts: queue.counts() };++}++++module.exports = { start, stop, kick, tick, abort, runNow, execute, status, scheduleInvariantScans, isStarted: () => started, _running: running };nnew file mode 100644@@@ -0,0 +1,91 @@++/**++ * OpenVibe.Media — content-type checks for v2 uploads (docs/object-model.md#content-types)++ *++ * 1. The declared type must suit the kind: vod -> video/* or audio/*, clip -> video/*, thumbnail /++ * screenshot / avatar -> image/* (not SVG). file and asset take any well-formed type.++ * 2. Bytes that /o/:id serves inline (image/*, video/*, audio/*, application/pdf) must look like what++ * their type says: the first bytes are checked against the container signatures below at++ * complete. A kind with a rule is always checked.++ * Anything else is served as an attachment with nosniff, so its declared type is taken as given.++ */++'use strict';++++const fs = require('fs');++++const TYPE_RE = /^[\w.+-]+\/[\w.+-]+$/;++const KIND_RULES = {++ vod: { re: /^(video|audio)\//, what: 'video/* or audio/*' },++ clip: { re: /^video\//, what: 'video/*' },++ thumbnail: { re: /^image\/(?!svg)/, what: 'image/* (not SVG)' },++ screenshot: { re: /^image\/(?!svg)/, what: 'image/* (not SVG)' },++ avatar: { re: /^image\/(?!svg)/, what: 'image/* (not SVG)' },++};++++function normalize(type) {++ const t = String(type || '').split(';')[0].trim().toLowerCase();++ return TYPE_RE.test(t) ? t : null;++}++++/** null when `type` suits `kind`, else the reason. */++function kindProblem(kind, type) {++ const rule = KIND_RULES[kind];++ if (!rule) return null;++ if (!type) return `a ${kind} needs a content type (${rule.what})`;++ return rule.re.test(type) ? null : `a ${kind} must be ${rule.what}, not ${type}`;++}++++/**++ * What the first bytes are: { family: 'image' | 'video' | 'av' | 'audio' | 'pdf', format } or null.++ * 'av' = a container that holds video, audio or both (Matroska/WebM, MP4/MOV, Ogg).++ */++function sniff(head) {++ const b = Buffer.isBuffer(head) ? head : Buffer.from(head || []);++ const at = (i, s) => b.length >= i + s.length && b.subarray(i, i + s.length).toString('latin1') === s;++ if (b.length >= 3 && b[0] === 0xff && b[1] === 0xd8 && b[2] === 0xff) return { family: 'image', format: 'jpeg' };++ if (at(0, '\x89PNG\r\n\x1a\n')) return { family: 'image', format: 'png' };++ if (at(0, 'GIF87a') || at(0, 'GIF89a')) return { family: 'image', format: 'gif' };++ if (at(0, 'RIFF') && at(8, 'WEBP')) return { family: 'image', format: 'webp' };++ if (at(0, 'BM')) return { family: 'image', format: 'bmp' };++ if (at(4, 'ftyp')) {++ const brand = b.subarray(8, 12).toString('latin1');++ if (/^(avif|avis|heic|heix|mif1|msf1)$/.test(brand)) return { family: 'image', format: brand };++ if (/^M4A |^M4B /.test(brand)) return { family: 'audio', format: 'm4a' };++ return { family: 'av', format: 'mp4' };++ }++ if (b.length >= 4 && b[0] === 0x1a && b[1] === 0x45 && b[2] === 0xdf && b[3] === 0xa3) return { family: 'av', format: 'matroska' };++ if (at(0, 'OggS')) return { family: 'av', format: 'ogg' };++ if (at(0, 'RIFF') && at(8, 'AVI ')) return { family: 'video', format: 'avi' };++ if (at(0, 'RIFF') && at(8, 'WAVE')) return { family: 'audio', format: 'wav' };++ if (at(0, 'fLaC')) return { family: 'audio', format: 'flac' };++ if (at(0, 'ID3') || (b.length >= 2 && b[0] === 0xff && (b[1] & 0xe0) === 0xe0)) return { family: 'audio', format: 'mp3' };++ if (b.length >= 4 && b[0] === 0x47 && b.length >= 189 && b[188] === 0x47) return { family: 'video', format: 'mpegts' };++ if (at(0, '%PDF-')) return { family: 'pdf', format: 'pdf' };++ return null;++}++++/** Is `type` served inline (so its bytes must match it)? */++function servedInline(type) {++ return /^(image\/|video\/|audio\/)/.test(type || '') || type === 'application/pdf';++}++++/** null when the bytes fit the declared type (or need no check), else the reason. */++function contentProblem(kind, type, head) {++ if (!KIND_RULES[kind] && !servedInline(type)) return null;++ const family = String(type || '').split('/')[0];++ const want = type === 'application/pdf' ? 'pdf' : family;++ const got = sniff(head);++ if (!got) return `the bytes are not a recognised ${want} format`;++ const ok = got.family === want || (got.family === 'av' && (want === 'video' || want === 'audio'));++ return ok ? null : `the bytes are ${got.format} (${got.family}), not ${type}`;++}++++function readHead(file, n = 512) {++ const fd = fs.openSync(file, 'r');++ try {++ const buf = Buffer.alloc(n);++ const got = fs.readSync(fd, buf, 0, n, 0);++ return buf.subarray(0, got);++ } finally { fs.closeSync(fd); }++}++++module.exports = { normalize, kindProblem, sniff, servedInline, contentProblem, readHead, KIND_RULES };nnew file mode 100644@@@ -0,0 +1,264 @@++/**++ * OpenVibe.Media — multipart uploads of native v2 objects (docs/object-model.md#multipart-uploads)++ *++ * A session belongs to one `uploading` object whose size is declared. Every part but the last is++ * exactly part_size bytes, the last is the rest, so the server knows each part's size up front and++ * refuses any other. Parts are stored under OBJECTS_PATH/.parts/<upload id>/<n> (written to a temp++ * file and renamed, so a dropped connection never leaves half a part) and recorded with their++ * sha256 in media_upload_parts. Re-sending a part replaces it. That makes a session resumable: after++ * a drop, GET the session, then send the parts in `missing`.++ *++ * Complete checks that every part is there (and matches the client's per-part sha256 when given),++ * concatenates them into OBJECTS_PATH/<app>/<id> while hashing the whole object, and then the route++ * runs the same checks as a single-part complete. Abort (or expiry after MEDIA_MULTIPART_TTL_HOURS)++ * deletes the parts; the object stays `uploading` so it can be sent again.++ */++'use strict';++++const fs = require('fs');++const path = require('path');++const crypto = require('crypto');++const { once } = require('events');++const { ids } = require('openvibe-contracts');++const db = require('../db/database');++const config = require('../config');++++const MB = 1024 * 1024;++const MAX_PARTS = 10000;++++function partsRoot() { return path.join(config.objects.path, '.parts'); }++function sessionDir(uploadId) { return path.join(partsRoot(), uploadId); }++function partFile(uploadId, n) { return path.join(sessionDir(uploadId), String(n)); }++++function getSession(uploadId) {++ return uploadId ? db.get('SELECT * FROM media_uploads WHERE id = ?', [String(uploadId)]) : null;++}++++function activeFor(objectId) {++ return db.get("SELECT * FROM media_uploads WHERE object_id = ? AND status IN ('active', 'completing') ORDER BY created_at DESC LIMIT 1", [objectId]);++}++++function listParts(uploadId) {++ return db.all('SELECT part_number, size_bytes, sha256, received_at FROM media_upload_parts WHERE upload_id = ? ORDER BY part_number', [uploadId]);++}++++/** Exact size of part n (1-based) of a session. */++function partSize(session, n) {++ if (n < session.parts_expected) return session.part_size;++ return session.total_size - session.part_size * (session.parts_expected - 1);++}++++function isExpired(session) {++ return Date.parse(String(session.expires_at).replace(' ', 'T') + 'Z') < Date.now();++}++++/** Part size for a total: the requested one within bounds, grown when the part count would pass 10,000. */++function choosePartSize(total, requested) {++ const min = config.objects.multipartMinPartMb * MB;++ const max = config.objects.multipartMaxPartMb * MB;++ let size = Number(requested) > 0 ? Math.floor(Number(requested)) : config.objects.multipartDefaultPartMb * MB;++ if (size < min) return { error: `part_size must be at least ${min} bytes` };++ if (size > max) return { error: `part_size must be at most ${max} bytes` };++ if (Math.ceil(total / size) > MAX_PARTS) size = Math.ceil(total / MAX_PARTS);++ if (size > max) return { error: 'the object is too large for multipart parts of the maximum size' };++ return { size };++}++++function freeBytes() {++ try { const st = fs.statfsSync(config.objects.path); return Number(st.bavail) * Number(st.bsize); } catch { return Infinity; }++}++++/**++ * Start a session for `obj` (uploading, size declared). An active session for the object is++ * replaced (its parts deleted). Returns the session row, or { error, status, code }.++ */++function initiate(obj, { partSize: requested } = {}) {++ const total = Number(obj.size_bytes) || 0;++ if (!total) return { status: 400, code: 'media.object.invalid', error: 'Declare size_bytes before a multipart upload' };++ if (total > config.objects.multipartMaxMb * MB) return { status: 413, code: 'media.object.too_large', error: `Multipart uploads are limited to ${config.objects.multipartMaxMb * MB} bytes` };++ const chosen = choosePartSize(total, requested);++ if (chosen.error) return { status: 400, code: 'media.object.invalid', error: chosen.error };++ // Parts and the assembled object both sit on disk until complete.++ const need = 2 * total + config.objects.uploadMinFreeMb * MB;++ const free = freeBytes();++ if (free < need) return { status: 507, code: 'media.storage.insufficient', error: `Not enough free space for this upload (${Math.floor(free / MB)} MB free)` };++ const prior = activeFor(obj.id);++ if (prior) abort(prior.id);++ const id = `mup_${ids.ulid()}`;++ const parts = Math.max(1, Math.ceil(total / chosen.size));++ db.run(`INSERT INTO media_uploads (id, object_id, app_id, part_size, total_size, parts_expected, expires_at)++ VALUES (?, ?, ?, ?, ?, ?, datetime('now', ?))`, [id, obj.id, obj.app_id, chosen.size, total, parts, `+${Math.max(1, config.objects.multipartTtlHours)} hours`]);++ fs.mkdirSync(sessionDir(id), { recursive: true });++ return getSession(id);++}++++/**++ * Stream one part to disk. Resolves { part_number, size_bytes, sha256 } or { status, code, error }.++ * `expectSha256` (X-Content-SHA256) must match when given.++ */++function receivePart(req, session, n, { expectSha256 = null } = {}) {++ const want = partSize(session, n);++ const tmp = `${partFile(session.id, n)}.${crypto.randomBytes(4).toString('hex')}.tmp`;++ fs.mkdirSync(sessionDir(session.id), { recursive: true });++ return new Promise((resolve) => {++ const hash = crypto.createHash('sha256');++ const out = fs.createWriteStream(tmp);++ let got = 0, done = false;++ const drop = () => { try { fs.unlinkSync(tmp); } catch { /* not written */ } };++ const finish = (r) => { if (done) return; done = true; resolve(r); };++ const fail = (r) => {++ if (done) return;++ const end = () => { drop(); finish(r); };++ if (out.closed) end(); else { out.once('close', end); out.destroy(); }++ };++ req.on('data', (c) => {++ if (done) return;++ got += c.length;++ if (got > want) { req.unpipe(out); req.resume(); return fail({ status: 413, code: 'media.upload.part_too_large', error: `Part ${n} is ${want} bytes` }); }++ hash.update(c);++ });++ req.on('aborted', () => fail({ status: 400, code: 'media.object.upload_interrupted', error: 'Upload did not complete' }));++ req.on('error', () => fail({ status: 400, code: 'media.object.upload_interrupted', error: 'Upload did not complete' }));++ out.on('error', (e) => fail({ status: 500, code: 'media.object.store_failed', error: e.message }));++ out.on('finish', () => {++ if (done) return;++ const sha = hash.digest('hex');++ if (got !== want) { drop(); return finish({ status: 400, code: 'media.upload.part_size_mismatch', error: `Part ${n} must be ${want} bytes, received ${got}` }); }++ if (expectSha256 && String(expectSha256).toLowerCase() !== sha) { drop(); return finish({ status: 400, code: 'media.upload.part_hash_mismatch', error: `Part ${n} does not match X-Content-SHA256` }); }++ // The session may have been completed or aborted while this part streamed in.++ const now = getSession(session.id);++ if (!now || now.status !== 'active') { drop(); return finish({ status: 409, code: 'media.upload.not_active', error: `The upload is ${now ? now.status : 'gone'}` }); }++ try {++ fs.renameSync(tmp, partFile(session.id, n));++ db.run(`INSERT INTO media_upload_parts (upload_id, part_number, size_bytes, sha256) VALUES (?, ?, ?, ?)++ ON CONFLICT(upload_id, part_number) DO UPDATE SET size_bytes = excluded.size_bytes, sha256 = excluded.sha256, received_at = CURRENT_TIMESTAMP`,++ [session.id, n, got, sha]);++ db.run('UPDATE media_uploads SET updated_at = CURRENT_TIMESTAMP WHERE id = ?', [session.id]);++ } catch (e) { drop(); return finish({ status: 500, code: 'media.object.store_failed', error: e.message }); }++ finish({ part_number: n, size_bytes: got, sha256: sha });++ });++ req.pipe(out);++ });++}++++/** Public shape of a session (never paths). */++function sessionPublic(session, { parts = true } = {}) {++ const out = {++ upload_id: session.id, object_id: session.object_id, status: session.status,++ part_size: session.part_size, total_size: session.total_size, parts_expected: session.parts_expected,++ expires_at: new Date(Date.parse(String(session.expires_at).replace(' ', 'T') + 'Z')).toISOString(),++ created_at: session.created_at,++ };++ if (parts) {++ const list = listParts(session.id);++ const have = new Set(list.map(p => p.part_number));++ out.parts = list;++ out.received_bytes = list.reduce((a, p) => a + p.size_bytes, 0);++ const missing = [];++ for (let n = 1; n <= session.parts_expected; n++) if (!have.has(n)) missing.push(n);++ out.missing = missing;++ }++ return out;++}++++/** active -> completing, unless someone else got there first. */++function beginComplete(session) {++ return db.run("UPDATE media_uploads SET status = 'completing', updated_at = CURRENT_TIMESTAMP WHERE id = ? AND status = 'active'", [session.id]).changes > 0;++}++function reopen(session) {++ db.run("UPDATE media_uploads SET status = 'active', updated_at = CURRENT_TIMESTAMP WHERE id = ? AND status = 'completing'", [session.id]);++}++++/**++ * Check the parts and concatenate them into `dest` (via a temp file), hashing the whole object.++ * `clientParts` [{ part_number, sha256 }] (optional) must match what was received.++ * Resolves { bytes, sha256 } or { status, code, error, missing? }.++ */++async function assemble(session, dest, clientParts = null) {++ const list = listParts(session.id);++ const byN = new Map(list.map(p => [p.part_number, p]));++ const missing = [];++ for (let n = 1; n <= session.parts_expected; n++) if (!byN.has(n) || !fs.existsSync(partFile(session.id, n))) missing.push(n);++ if (missing.length) return { status: 409, code: 'media.upload.parts_missing', error: `${missing.length} part(s) missing`, missing };++ if (Array.isArray(clientParts)) {++ for (const c of clientParts) {++ const p = byN.get(Number(c && c.part_number));++ if (!p) return { status: 400, code: 'media.upload.invalid_part', error: `No part ${c && c.part_number}` };++ if (c.sha256 && String(c.sha256).toLowerCase() !== p.sha256) return { status: 400, code: 'media.upload.part_hash_mismatch', error: `Part ${p.part_number} differs from the one received` };++ }++ }++ fs.mkdirSync(path.dirname(dest), { recursive: true });++ const tmp = `${dest}.${crypto.randomBytes(4).toString('hex')}.assembling`;++ const hash = crypto.createHash('sha256');++ const out = fs.createWriteStream(tmp);++ let bytes = 0;++ try {++ let failed = null;++ out.on('error', (e) => { failed = e; });++ for (let n = 1; n <= session.parts_expected; n++) {++ for await (const chunk of fs.createReadStream(partFile(session.id, n), { highWaterMark: 1024 * 1024 })) {++ if (failed) throw failed;++ hash.update(chunk);++ bytes += chunk.length;++ if (!out.write(chunk)) await once(out, 'drain');++ }++ }++ await new Promise((resolve, reject) => { out.end((err) => (err || failed ? reject(err || failed) : resolve())); });++ } catch (err) {++ out.destroy();++ try { fs.unlinkSync(tmp); } catch { /* none */ }++ return { status: 500, code: 'media.object.store_failed', error: err.message };++ }++ if (bytes !== session.total_size) {++ try { fs.unlinkSync(tmp); } catch { /* none */ }++ return { status: 400, code: 'media.object.size_mismatch', error: `Assembled ${bytes} bytes, declared ${session.total_size}` };++ }++ fs.renameSync(tmp, dest);++ return { bytes, sha256: hash.digest('hex') };++}++++function finish(session) {++ db.run("UPDATE media_uploads SET status = 'completed', completed_at = CURRENT_TIMESTAMP, updated_at = CURRENT_TIMESTAMP WHERE id = ?", [session.id]);++ removeParts(session.id);++}++++function removeParts(uploadId) {++ try { fs.rmSync(sessionDir(uploadId), { recursive: true, force: true }); } catch { /* best effort */ }++ db.run('DELETE FROM media_upload_parts WHERE upload_id = ?', [uploadId]);++}++++/** Delete a session's parts; the object stays uploading. */++function abort(uploadId, status = 'aborted') {++ db.run("UPDATE media_uploads SET status = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ? AND status IN ('active', 'completing')", [status, uploadId]);++ removeParts(uploadId);++}++++/**++ * Expire sessions past their expiry (incomplete multipart uploads) and remove part directories that++ * no session owns. Returns { expired, orphan_dirs }.++ */++function purgeExpired() {++ let expired = 0, orphans = 0;++ for (const s of db.all("SELECT id FROM media_uploads WHERE status IN ('active', 'completing') AND expires_at < datetime('now')")) {++ abort(s.id, 'expired');++ expired++;++ }++ let dirs = [];++ try { dirs = fs.readdirSync(partsRoot()); } catch { /* none yet */ }++ for (const d of dirs) {++ const s = getSession(d);++ if (s && ['active', 'completing'].includes(s.status)) continue;++ try { fs.rmSync(path.join(partsRoot(), d), { recursive: true, force: true }); orphans++; } catch { /* next time */ }++ }++ return { expired, orphan_dirs: orphans };++}++++/** Sessions that are still open (reconciliation: incomplete multipart uploads). */++function openSessions() {++ return db.all("SELECT u.*, (SELECT COUNT(*) FROM media_upload_parts p WHERE p.upload_id = u.id) AS parts_received FROM media_uploads u WHERE status IN ('active', 'completing') ORDER BY created_at");++}++++module.exports = {++ MAX_PARTS, getSession, activeFor, listParts, partSize, isExpired, choosePartSize,++ initiate, receivePart, sessionPublic, beginComplete, reopen, assemble, finish, abort, purgeExpired, openSessions, partFile,++};@@@ -17,6 +17,8 @@* orphan_location location rows whose object does not exist* deleted_publicly_reachable deleted object still served (legacy row, or a thumbnail of it)* missing_projection inherited rows with no object yet (run the backfill)++ * incomplete_multipart multipart sessions still open past their expiry (the hourly purge++ * removes their parts; one listed here means the purge is not running)*/'use strict';@@@ -27,7 +29,7 @@ const model = require('./model');const MB = 1024 * 1024;const ISSUES = ['no_canonical_location', 'canonical_missing_replica_present', 'no_present_copy', 'missing_local_file', 'remote_missing',-- 'size_mismatch', 'hash_mismatch', 'orphan_location', 'deleted_publicly_reachable', 'missing_projection'];++ 'size_mismatch', 'hash_mismatch', 'orphan_location', 'deleted_publicly_reachable', 'missing_projection', 'incomplete_multipart'];/** Real provider: undefined when the provider is not configured here (cannot verify), null on 404. */async function defaultHead(provider, key) {@@@ -181,6 +183,11 @@ async function reconcile({ verify = false, hash = false, head = defaultHead, has}}++ for (const u of db.all(`SELECT u.*, (SELECT COUNT(*) FROM media_upload_parts p WHERE p.upload_id = u.id) AS parts_received FROM media_uploads u++ WHERE u.status IN ('active', 'completing') AND u.expires_at < datetime('now')${appId ? ' AND u.app_id = ?' : ''}`, appId ? [appId] : [])) {++ issue('incomplete_multipart', { upload_id: u.id, object_id: u.object_id, parts_received: u.parts_received, parts_expected: u.parts_expected, expires_at: u.expires_at });++ }++report.total_issues = ISSUES.reduce((n, k) => n + report.issues[k].count, 0);return report;}@@@ -1,9 +1,16 @@/*** OpenVibe.Media — Object API v2 (mounted at /api/v2/:app/objects; docs/object-model.md)*-- * POST / init upload → { id, object, upload: { method, url, token, expires_at, max_bytes } }-- * PUT /:id/content the bytes (single part; sha256 computed; size + quota checked)-- * POST /:id/complete verify (expected hash, invariant) → lifecycle ready++ * POST / init upload → { id, object, upload: { method, url, token, expires_at, max_bytes, … } }++ * (objects over MEDIA_OBJECT_MAX_MB, or { multipart: true }: a multipart session)++ * POST /:id/upload-url a fresh presigned single-PUT URL (scoped to tenant, object and size; ?ttl)++ * PUT /:id/content the bytes (single part; sha256 computed; size + quota + content type checked)++ * POST /:id/complete verify (expected hash, invariant, quota, content bytes) → lifecycle ready++ * POST /:id/multipart start a multipart session { part_size } → upload_id, token++ * GET /:id/multipart/:uploadId session: parts received (sha256), `missing` (resume)++ * PUT /:id/multipart/:uploadId/parts/:n one part (exact size; X-Content-SHA256 optional)++ * POST /:id/multipart/:uploadId/complete assemble + the same checks as complete → ready++ * DELETE /:id/multipart/:uploadId abort (parts deleted; the object stays uploading)* GET /:id metadata (id = med_… or legacy:<app>:<kind>:<id>)* GET / cursor list (?kind&visibility&status&owner&user_id&include_deleted&limit&cursor)* DELETE /:id soft delete (bytes kept MEDIA_DELETE_RETENTION_DAYS); 409 when held@@@ -18,8 +25,8 @@* (:app = prj_<ULID>) take their project's app tokens only (server/auth.js); their sandbox* objects are served only through signed URLs, whatever their visibility. X-OV-Subject names the* owner (usr_…); X-OV-User-Id (app key only) acts as one of the app's users. The-- * content PUT and complete also accept the upload token handed out at init, so a-- * browser can send the bytes directly.++ * content PUT and complete also accept the upload token handed out at init (a presigned++ * URL), and the multipart routes the session token, so a browser can send the bytes directly.** Public bytes: GET /o/:id (publicRouter) — public/unlisted objects openly,* private ones only with a valid ?exp&sig from /download.@@@ -36,6 +43,8 @@ const config = require('../config');const model = require('./model');const invariant = require('./invariant');const signing = require('./signing');++const ctypes = require('./content-type');++const multipart = require('./multipart');const { tenantAuth, tenantCors, tenantPath } = require('../auth');const { announce } = require('../webhooks');@@@ -73,22 +82,77 @@ function load(req, res) {return obj;}--/** Upload token (from init) or the usual tenant credential with media.object.upload. */--function contentAuth(req, res, next) {-- const token = req.query.token || req.headers['x-upload-token'];-- if (!token) return upload(req, res, next);-- const id = String(req.params.id || '');-- if (!signing.verifyUploadToken(id, token)) return problem(res, 401, 'media.upload_token.invalid', 'Upload token is invalid or expired');-- // The token names the object; its tenant must be the one the URL addresses (a developer-project-- // tenant is addressed by its project id, whichever of its production/sandbox rows holds the object).-- const obj = model.getObject(id);++/** The token-bearing request's object and tenant; the URL must address that tenant. */++function tokenTenant(req, res, obj) {++ // A developer-project tenant is addressed by its project id, whichever of its production/sandbox rows holds the object.const app = obj ? db.getApp(obj.app_id) : null;-- if (!app || tenantPath(app) !== String(req.params.app || '')) return res.status(404).json({ error: 'Unknown app' });++ if (!app || tenantPath(app) !== String(req.params.app || '')) { res.status(404).json({ error: 'Unknown app' }); return false; }req.appId = app.app_id;req.appPath = tenantPath(app);req.appRow = app;req.authType = 'upload_token';-- next();++ return true;++}++++/** Upload token (a presigned URL from init or /upload-url) or the usual tenant credential with media.object.upload. */++function contentAuth(req, res, next) {++ const token = req.query.token || req.headers['x-upload-token'];++ if (!token) return upload(req, res, next);++ const id = String(req.params.id || '');++ const obj = model.getObject(id);++ // Scoped to tenant + object + size: a token for another object, tenant or size never verifies.++ const size = obj && Number(obj.size_bytes) > 0 ? Number(obj.size_bytes) : 0;++ if (!obj || !signing.verifyUploadToken(id, token, { tenant: obj.app_id, size })) return problem(res, 401, 'media.upload_token.invalid', 'Upload token is invalid or expired');++ if (tokenTenant(req, res, obj)) next();++}++++/** Multipart session token (from POST /:id/multipart) or the usual tenant credential with `capability`. */++function multipartAuth(capability) {++ const fallback = tenantAuth({ capability });++ return (req, res, next) => {++ const token = req.query.token || req.headers['x-upload-token'];++ if (!token) return fallback(req, res, next);++ const obj = model.getObject(String(req.params.id || ''));++ const session = multipart.getSession(String(req.params.uploadId || ''));++ const ok = obj && session && session.object_id === obj.id && signing.verifyMultipartToken(token, {++ tenant: obj.app_id, objectId: obj.id, uploadId: session.id, totalSize: session.total_size,++ });++ if (!ok) return problem(res, 401, 'media.upload_token.invalid', 'Upload token is invalid or expired');++ if (tokenTenant(req, res, obj)) next();++ };++}++++function apiBase(req) {++ return `${config.publicUrl}/api/v2/${encodeURIComponent(req.appPath || req.appId)}/objects`;++}++++/** A presigned single-PUT URL for this object: { method, url, token, expires_at, max_bytes, content_type }. */++function presignedPut(req, obj, ttlS) {++ const size = Number(obj.size_bytes) > 0 ? Number(obj.size_bytes) : 0;++ const tok = signing.uploadToken(obj.id, ttlS, { tenant: obj.app_id, size });++ return {++ method: 'PUT',++ url: `${apiBase(req)}/${obj.id}/content?token=${encodeURIComponent(tok.token)}`,++ token: tok.token,++ expires_at: tok.expires_at,++ max_bytes: size || config.objects.maxUploadMb * MB,++ content_type: obj.mime_type || null,++ };++}++++/** Public shape of a multipart session plus what a client needs to send it (URLs carry the session token). */++function multipartPublic(req, obj, session, { parts = true } = {}) {++ const tok = signing.multipartToken({ tenant: obj.app_id, objectId: obj.id, uploadId: session.id, totalSize: session.total_size, expiresAt: Date.parse(String(session.expires_at).replace(' ', 'T') + 'Z') });++ const u = `${apiBase(req)}/${obj.id}/multipart/${session.id}`;++ const q = `token=${encodeURIComponent(tok.token)}`;++ return {++ ...multipart.sessionPublic(session, { parts }),++ token: tok.token,++ part_url_template: `${u}/parts/{part_number}?${q}`,++ status_url: `${u}?${q}`,++ complete_url: `${u}/complete?${q}`,++ abort_url: `${u}?${q}`,++ };}function sanitizeFilename(name) {@@@ -138,12 +202,23 @@ async function putContent(req, res) {if (obj.lifecycle_status !== 'uploading') { req.resume(); return problem(res, 409, 'media.object.not_uploading', `Object is ${obj.lifecycle_status}`); }const declared = Number(obj.size_bytes) > 0 ? Number(obj.size_bytes) : null;-- const cap = declared || config.objects.maxUploadMb * MB;++ const single = config.objects.maxUploadMb * MB;++ if (declared && declared > single) {++ res.set('Connection', 'close'); req.resume();++ return problem(res, 413, 'media.object.too_large', `Objects over ${single} bytes are uploaded in parts: POST /${obj.id}/multipart`);++ }++ if (multipart.activeFor(obj.id)) { req.resume(); return problem(res, 409, 'media.upload.multipart_active', 'A multipart upload is open for this object: complete or abort it first'); }++ const cap = declared || single;const len = Number(req.headers['content-length']);if (Number.isFinite(len) && len > cap) {res.set('Connection', 'close'); req.resume();return problem(res, 413, 'media.object.too_large', `At most ${cap} bytes`);}++ // No type declared at init: the request's Content-Type becomes it, and must suit the kind.++ const headerType = ctypes.normalize(req.headers['content-type']);++ const effectiveType = obj.mime_type || headerType || 'application/octet-stream';++ const typeProblem = ctypes.kindProblem(obj.kind, obj.mime_type || headerType);++ if (typeProblem) { req.resume(); return problem(res, 415, 'media.object.unsupported_type', typeProblem); }const tmpDir = path.join(config.objects.path, '.tmp');fs.mkdirSync(tmpDir, { recursive: true });@@@ -163,11 +238,9 @@ async function putContent(req, res) {const dest = model.objectFilePath(obj);fs.mkdirSync(path.dirname(dest), { recursive: true });fs.renameSync(tmp, dest);-- const contentType = String(req.headers['content-type'] || '').split(';')[0].trim().toLowerCase();db.getDb().transaction(() => {model.updateObject(obj.id, {-- size_bytes: got.bytes, content_hash: got.sha256,-- mime_type: obj.mime_type || (/^[\w.+-]+\/[\w.+-]+$/.test(contentType) ? contentType : 'application/octet-stream'),++ size_bytes: got.bytes, content_hash: got.sha256, mime_type: effectiveType,canonical_provider: 'local', canonical_key: dest,});model.upsertLocation(obj.id, { provider: 'local', key: dest, state: 'present', size_bytes: got.bytes, checksum: got.sha256, verified: true });@@@ -198,7 +271,10 @@ router.post('/', upload, (req, res) => {if (!Number.isInteger(size) || size < 1) return problem(res, 400, 'media.object.invalid', 'size_bytes must be a positive integer');}const max = config.objects.maxUploadMb * MB;-- if (size > max) return problem(res, 413, 'media.object.too_large', `Single-part uploads are limited to ${max} bytes`);++ const wantParts = b.multipart === true || b.multipart === 'true';++ if (wantParts && !size) return problem(res, 400, 'media.object.invalid', 'A multipart upload needs size_bytes');++ if (!wantParts && size > max) return problem(res, 413, 'media.object.too_large', `Single-part uploads are limited to ${max} bytes; send multipart: true for larger objects`);++ if (wantParts && size > config.objects.multipartMaxMb * MB) return problem(res, 413, 'media.object.too_large', `Objects are limited to ${config.objects.multipartMaxMb * MB} bytes`);if (size && invariant.wouldViolate({ kind, visibility, size_bytes: size })) {return problem(res, 422, 'media.invariant.public_object_too_large', `Public playback objects are limited to ${config.objects.publicMaxMb} MB`);}@@@ -211,6 +287,11 @@ router.post('/', upload, (req, res) => {if (!quotaCheck(req, res, size)) return;const mime = b.mime_type && /^[\w.+-]+\/[\w.+-]+$/.test(String(b.mime_type)) ? String(b.mime_type).toLowerCase() : null;++ if (mime) {++ const why = ctypes.kindProblem(kind, mime);++ if (why) return problem(res, 415, 'media.object.unsupported_type', why);++ }++ if (b.upload_ttl != null && !(Number(b.upload_ttl) >= 60 && Number(b.upload_ttl) <= 86400)) return problem(res, 400, 'media.object.invalid', 'upload_ttl must be 60-86400 seconds');const userId = req.authType === 'user' ? req.userId : (Number.isInteger(Number(b.user_id)) && Number(b.user_id) > 0 ? Number(b.user_id) : null);const metadata = { ...(b.metadata || {}) };const filename = sanitizeFilename(b.filename);@@@ -222,17 +303,24 @@ router.post('/', upload, (req, res) => {app_id: req.appId, namespace: req.appId, kind, owner_subject: subject, owner_app: req.appId, owner_user_id: userId,visibility, lifecycle_status: 'uploading', mime_type: mime, size_bytes: size, metadata,});-- const tok = signing.uploadToken(id);++ const obj = model.getObject(id);++ let session = null;++ if (wantParts) {++ session = multipart.initiate(obj, { partSize: b.part_size });++ if (session.error) {++ db.run('DELETE FROM media_objects WHERE id = ?', [id]); // nothing was stored for it++ return problem(res, session.status, session.code, session.error);++ }++ }++ const single = !wantParts ? presignedPut(req, obj, b.upload_ttl) : null;res.status(201).json({id,-- object: model.objectPublic(model.getObject(id)),++ object: model.objectPublic(obj),upload: {-- method: 'PUT',-- url: `${config.publicUrl}/api/v2/${encodeURIComponent(req.appPath || req.appId)}/objects/${id}/content?token=${encodeURIComponent(tok.token)}`,-- token: tok.token,-- expires_at: tok.expires_at,-- max_bytes: size || max,-- complete_url: `${config.publicUrl}/api/v2/${encodeURIComponent(req.appPath || req.appId)}/objects/${id}/complete`,++ ...(single || { method: 'multipart', url: null, token: null, expires_at: null, max_bytes: size, content_type: mime }),++ complete_url: `${apiBase(req)}/${id}/complete`,++ multipart_url: `${apiBase(req)}/${id}/multipart`,++ ...(session ? { multipart: multipartPublic(req, obj, session, { parts: false }) } : {}),},});} catch (err) {@@@ -241,6 +329,45 @@ router.post('/', upload, (req, res) => {}});++/**++ * The checks every upload ends with (single PUT and multipart): expected sha256, the public-size++ * invariant, the quota, and whether the bytes are what their type says; then the ready transition,++ * its invariant row and the media.object.uploaded event commit together (the webhook follows).++ */++function completeUpload(req, res, obj, { expectedHash = null } = {}) {++ const loc = model.listLocations(obj.id).find(l => l.provider === 'local' && l.state === 'present');++ if (!loc || !fs.existsSync(loc.key)) return problem(res, 409, 'media.object.no_content', 'PUT the content first');++ const md = model.parseJson(obj.metadata, {});++ const expected = String(expectedHash || md.expected_sha256 || '').toLowerCase();++ if (expected && expected !== obj.content_hash) {++ return problem(res, 422, 'media.object.hash_mismatch', 'Content hash does not match', { expected, actual: obj.content_hash });++ }++ if (invariant.wouldViolate(obj)) {++ return problem(res, 422, 'media.invariant.public_object_too_large', `Public playback objects are limited to ${config.objects.publicMaxMb} MB`);++ }++ if (!quotaCheck(req, res, Number(obj.size_bytes) || 0, obj.id)) return;++ const typeWhy = ctypes.kindProblem(obj.kind, obj.mime_type) || ctypes.contentProblem(obj.kind, obj.mime_type, ctypes.readHead(loc.key));++ if (typeWhy) return problem(res, 415, 'media.object.content_mismatch', typeWhy);++ const { data: body } = announce(req.appId, 'media.object.uploaded', {++ change: () => {++ model.updateObject(obj.id, { lifecycle_status: 'ready' });++ invariant.record(model.getObject(obj.id));++ },++ payload: () => model.objectPublic(model.getObject(obj.id)),++ });++ return res.json(body);++}++++/** The object named in the URL, writable by this caller, native and still uploading (else answered). */++function loadUploading(req, res) {++ const obj = load(req, res);++ if (!obj) return null;++ if (!canWrite(req, obj)) { problem(res, 403, 'media.object.forbidden', 'Not your object'); return null; }++ if (obj.legacy_ref) { problem(res, 409, 'media.object.legacy_managed', 'This object is written through the v1 API'); return null; }++ if (obj.lifecycle_status !== 'uploading') { problem(res, 409, 'media.object.not_uploading', `Object is ${obj.lifecycle_status}`); return null; }++ return obj;++}++router.post('/:id/complete', contentAuth, tenantCors, (req, res) => {try {const obj = load(req, res);@@@ -248,33 +375,119 @@ router.post('/:id/complete', contentAuth, tenantCors, (req, res) => {if (!canWrite(req, obj)) return problem(res, 403, 'media.object.forbidden', 'Not your object');if (obj.lifecycle_status === 'ready' && !obj.legacy_ref) return res.json(model.objectPublic(obj));if (obj.legacy_ref || obj.lifecycle_status !== 'uploading') return problem(res, 409, 'media.object.not_uploading', `Object is ${obj.lifecycle_status}`);-- const loc = model.listLocations(obj.id).find(l => l.provider === 'local' && l.state === 'present');-- if (!loc || !fs.existsSync(loc.key)) return problem(res, 409, 'media.object.no_content', 'PUT the content first');-- const md = model.parseJson(obj.metadata, {});-- const expected = String((req.body && req.body.content_hash) || md.expected_sha256 || '').toLowerCase();-- if (expected && expected !== obj.content_hash) {-- return problem(res, 422, 'media.object.hash_mismatch', 'Content hash does not match', { expected, actual: obj.content_hash });-- }-- if (invariant.wouldViolate(obj)) {-- return problem(res, 422, 'media.invariant.public_object_too_large', `Public playback objects are limited to ${config.objects.publicMaxMb} MB`);-- }-- if (!quotaCheck(req, res, Number(obj.size_bytes) || 0, obj.id)) return;-- // The ready transition, its invariant row and the media.object.uploaded event commit-- // together; the webhook follows the commit.-- const { data: body } = announce(req.appId, 'media.object.uploaded', {-- change: () => {-- model.updateObject(obj.id, { lifecycle_status: 'ready' });-- invariant.record(model.getObject(obj.id));-- },-- payload: () => model.objectPublic(model.getObject(obj.id)),-- });-- res.json(body);++ completeUpload(req, res, obj, { expectedHash: req.body && req.body.content_hash });} catch (err) {console.error('[Objects] complete error:', err.message);problem(res, 500, 'media.object.complete_failed', 'Failed to complete upload');}});++// ── Presigned single-PUT URL ─────────────────────────────────++++router.post('/:id/upload-url', upload, (req, res) => {++ const obj = loadUploading(req, res);++ if (!obj) return;++ const ttl = req.body && req.body.ttl != null ? Number(req.body.ttl) : (req.query.ttl != null ? Number(req.query.ttl) : undefined);++ if (ttl !== undefined && !(ttl >= 60 && ttl <= 86400)) return problem(res, 400, 'media.object.invalid', 'ttl must be 60-86400 seconds');++ if (Number(obj.size_bytes) > config.objects.maxUploadMb * MB) return problem(res, 413, 'media.object.too_large', 'This object is uploaded in parts: POST /multipart');++ res.json(presignedPut(req, obj, ttl));++});++++// ── Multipart uploads ────────────────────────────────────────++++const mpRead = multipartAuth('media.object.read');++const mpWrite = multipartAuth('media.object.upload');++++/** The session named in the URL, belonging to the object, still open (else answered). */++function loadSession(req, res, obj, { open = true } = {}) {++ const session = multipart.getSession(String(req.params.uploadId || ''));++ if (!session || session.object_id !== obj.id) { problem(res, 404, 'media.upload.not_found', 'No such upload for this object'); return null; }++ if (open && session.status === 'active' && multipart.isExpired(session)) multipart.abort(session.id, 'expired');++ const now = multipart.getSession(session.id);++ if (open && now.status !== 'active') { problem(res, 409, 'media.upload.not_active', `The upload is ${now.status}`); return null; }++ return now;++}++++router.post('/:id/multipart', upload, (req, res) => {++ try {++ const obj = loadUploading(req, res);++ if (!obj) return;++ const b = req.body || {};++ let current = obj;++ if (!(Number(obj.size_bytes) > 0)) {++ // Created without a size: declare it now (the same checks init makes).++ const size = Number(b.size_bytes);++ if (!Number.isInteger(size) || size < 1) return problem(res, 400, 'media.object.invalid', 'size_bytes (a positive integer) is required');++ if (size > config.objects.multipartMaxMb * MB) return problem(res, 413, 'media.object.too_large', `Objects are limited to ${config.objects.multipartMaxMb * MB} bytes`);++ if (invariant.wouldViolate({ kind: obj.kind, visibility: obj.visibility, size_bytes: size })) {++ return problem(res, 422, 'media.invariant.public_object_too_large', `Public playback objects are limited to ${config.objects.publicMaxMb} MB`);++ }++ if (!quotaCheck(req, res, size, obj.id)) return;++ model.updateObject(obj.id, { size_bytes: size });++ current = model.getObject(obj.id);++ } else if (b.size_bytes != null && Number(b.size_bytes) !== Number(obj.size_bytes)) {++ return problem(res, 400, 'media.object.size_mismatch', `The object declares ${obj.size_bytes} bytes`);++ }++ const session = multipart.initiate(current, { partSize: b.part_size });++ if (session.error) return problem(res, session.status, session.code, session.error);++ res.status(201).json(multipartPublic(req, current, session, { parts: false }));++ } catch (err) {++ console.error('[Objects] multipart init error:', err.message);++ problem(res, 500, 'media.upload.init_failed', 'Failed to start the upload');++ }++});++++router.get('/:id/multipart/:uploadId', mpRead, (req, res) => {++ const obj = load(req, res);++ if (!obj) return;++ const session = loadSession(req, res, obj, { open: false });++ if (!session) return;++ if (session.status === 'active' && multipart.isExpired(session)) multipart.abort(session.id, 'expired');++ res.json(multipart.sessionPublic(multipart.getSession(session.id)));++});++++router.post('/:id/multipart/:uploadId/complete', mpWrite, tenantCors, async (req, res) => {++ let session = null;++ try {++ const obj = loadUploading(req, res);++ if (!obj) return;++ session = loadSession(req, res, obj);++ if (!session) return;++ if (!multipart.beginComplete(session)) return problem(res, 409, 'media.upload.not_active', 'The upload is being completed already');++ const b = req.body || {};++ const dest = model.objectFilePath(obj);++ const got = await multipart.assemble(session, dest, Array.isArray(b.parts) ? b.parts : null);++ if (got.error) {++ multipart.reopen(session);++ return problem(res, got.status, got.code, got.error, got.missing ? { missing: got.missing } : undefined);++ }++ db.getDb().transaction(() => {++ model.updateObject(obj.id, {++ size_bytes: got.bytes, content_hash: got.sha256, mime_type: obj.mime_type || 'application/octet-stream',++ canonical_provider: 'local', canonical_key: dest,++ });++ model.upsertLocation(obj.id, { provider: 'local', key: dest, state: 'present', size_bytes: got.bytes, checksum: got.sha256, verified: true });++ })();++ multipart.finish(session);++ completeUpload(req, res, model.getObject(obj.id), { expectedHash: b.content_hash });++ } catch (err) {++ if (session) multipart.reopen(session);++ console.error('[Objects] multipart complete error:', err.message);++ if (!res.headersSent) problem(res, 500, 'media.object.complete_failed', 'Failed to complete upload');++ }++});++++router.delete('/:id/multipart/:uploadId', mpWrite, (req, res) => {++ const obj = load(req, res);++ if (!obj) return;++ if (!canWrite(req, obj)) return problem(res, 403, 'media.object.forbidden', 'Not your object');++ const session = loadSession(req, res, obj, { open: false });++ if (!session) return;++ if (session.status === 'completed') return problem(res, 409, 'media.upload.not_active', 'The upload is completed');++ multipart.abort(session.id);++ res.json(multipart.sessionPublic(multipart.getSession(session.id), { parts: false }));++});++router.get('/', read, (req, res) => {try {const q = req.query;@@@ -419,7 +632,37 @@ publicRouter.get('/:id', async (req, res) => {}});++/** PUT /:id/multipart/:uploadId/parts/:n (raw body; mounted ahead of the JSON body parser). */++async function putPart(req, res) {++ const obj = load(req, res);++ if (!obj) { req.resume(); return; }++ if (!canWrite(req, obj)) { req.resume(); return problem(res, 403, 'media.object.forbidden', 'Not your object'); }++ if (obj.lifecycle_status !== 'uploading') { req.resume(); return problem(res, 409, 'media.object.not_uploading', `Object is ${obj.lifecycle_status}`); }++ const session = loadSession(req, res, obj);++ if (!session) { req.resume(); return; }++ const n = Number(req.params.n);++ if (!Number.isInteger(n) || n < 1 || n > session.parts_expected) {++ req.resume();++ return problem(res, 400, 'media.upload.invalid_part', `part_number must be 1-${session.parts_expected}`);++ }++ const want = multipart.partSize(session, n);++ const len = Number(req.headers['content-length']);++ if (Number.isFinite(len) && len !== want) {++ res.set('Connection', 'close'); req.resume();++ return problem(res, len > want ? 413 : 400, len > want ? 'media.upload.part_too_large' : 'media.upload.part_size_mismatch', `Part ${n} must be ${want} bytes`);++ }++ const expect = req.headers['x-content-sha256'];++ if (expect != null && !/^[a-f0-9]{64}$/i.test(String(expect))) { req.resume(); return problem(res, 400, 'media.upload.invalid_part', 'X-Content-SHA256 must be a sha256 hex digest'); }++ const got = await multipart.receivePart(req, session, n, { expectSha256: expect || null });++ if (got.error) {++ if (got.status === 413) res.set('Connection', 'close');++ return problem(res, got.status, got.code, got.error);++ }++ res.json(got);++}++module.exports = router;module.exports.contentHandlers = [contentAuth, tenantCors, putContent];++module.exports.partHandlers = [mpWrite, tenantCors, putPart];module.exports.publicRouter = publicRouter;module.exports.INLINE = INLINE;@@@ -3,7 +3,14 @@** HMAC-SHA256 over (purpose, object id, expiry) with MEDIA_SIGNING_SECRET.* Purposes are separate so a download signature can never be replayed as an-- * upload token or the other way round. Without the env secret a random++ * upload token or the other way round.++ *++ * Upload tokens (presigned PUT URLs) are scoped to the tenant, the object and its size:++ * "v2.<exp>.<size>.<mac>" over (put2, tenant, object id, size, exp); size 0 = "up to the++ * single-part limit" (the object was created without a declared size). Tokens of the earlier++ * form "<exp>.<mac>" (object only) are still accepted until they expire. Multipart sessions++ * get one token for all their parts: "mp1.<exp>.<mac>" over (mpart, tenant, object id,++ * upload id, total size, exp). Without the env secret a random* per-process secret is used (and a warning logged): links then stop working* at the next restart, which is safe but not what production wants.*/@@@ -67,18 +74,70 @@ function verifyFile(key, exp, sig) {return safeEqual(sig, mac('getf', `file:${key}`, e));}--/** Token for PUT /api/v2/:app/objects/:id/content: "<exp>.<mac>". */--function uploadToken(objectId, ttlS = config.objects.uploadTokenTtlS) {-- const exp = nowS() + Math.max(60, Number(ttlS) || 3600);-- return { token: `${exp}.${mac('put', objectId, exp)}`, expires_at: new Date(exp * 1000).toISOString() };++function macOf(parts) {++ return crypto.createHmac('sha256', secret()).update(parts.map(String).join('\n')).digest('base64url');++}++++/** Upload token lifetime: 60 s to 24 h (default MEDIA_UPLOAD_TOKEN_TTL_S). */++function uploadTtl(ttlS) {++ return Math.min(86400, Math.max(60, Number(ttlS) || config.objects.uploadTokenTtlS || 3600));++}++++/**++ * Token for PUT /api/v2/:app/objects/:id/content (a presigned single-PUT URL carries it as ?token=):++ * "v2.<exp>.<size>.<mac>" scoped to tenant + object + size.++ */++function uploadToken(objectId, ttlS = config.objects.uploadTokenTtlS, { tenant = '', size = 0 } = {}) {++ const exp = nowS() + uploadTtl(ttlS);++ const n = Math.max(0, Math.floor(Number(size) || 0));++ return { token: `v2.${exp}.${n}.${macOf(['put2', tenant, objectId, n, exp])}`, expires_at: new Date(exp * 1000).toISOString(), size: n };++}++++/**++ * Check an upload token's signature for this object and tenant. Returns { ok, size } where size is++ * the byte count the token is scoped to (signed, so it cannot be altered): 0 = up to the single-part++ * limit, null = an earlier-form token (object only).++ */++function checkUploadToken(objectId, token, { tenant = '' } = {}) {++ const t = String(token || '');++ const v2 = /^v2\.(\d+)\.(\d+)\.([A-Za-z0-9_-]+)$/.exec(t);++ if (v2) {++ const exp = Number(v2[1]), n = Number(v2[2]);++ if (exp < nowS()) return { ok: false };++ return { ok: safeEqual(v2[3], macOf(['put2', tenant, objectId, n, exp])), size: n };++ }++ const m = /^(\d+)\.([A-Za-z0-9_-]+)$/.exec(t);++ if (!m) return { ok: false };++ const exp = Number(m[1]);++ if (exp < nowS()) return { ok: false };++ return { ok: safeEqual(m[2], mac('put', objectId, exp)), size: null };++}++++/**++ * Valid for this object, in this tenant, for an object of `size` bytes: a token scoped to a size++ * is valid only while the object declares exactly that size (a size-0 token: any size up to the limit).++ */++function verifyUploadToken(objectId, token, { tenant = '', size } = {}) {++ const r = checkUploadToken(objectId, token, { tenant });++ if (!r.ok) return false;++ return size === undefined || r.size === null || r.size === 0 || r.size === Number(size);++}++++/** One token for every part (and the status, complete and abort calls) of a multipart session. */++function multipartToken({ tenant, objectId, uploadId, totalSize, expiresAt }) {++ const exp = Math.floor(new Date(expiresAt).getTime() / 1000);++ return { token: `mp1.${exp}.${macOf(['mpart', tenant, objectId, uploadId, totalSize, exp])}`, expires_at: new Date(exp * 1000).toISOString() };}--function verifyUploadToken(objectId, token) {-- const m = /^(\d+)\.([A-Za-z0-9_-]+)$/.exec(String(token || ''));++function verifyMultipartToken(token, { tenant, objectId, uploadId, totalSize }) {++ const m = /^mp1\.(\d+)\.([A-Za-z0-9_-]+)$/.exec(String(token || ''));if (!m) return false;const exp = Number(m[1]);if (exp < nowS()) return false;-- return safeEqual(m[2], mac('put', objectId, exp));++ return safeEqual(m[2], macOf(['mpart', tenant, objectId, uploadId, totalSize, exp]));}--module.exports = { signedDownloadUrl, verifyDownload, signedFileUrl, verifyFile, uploadToken, verifyUploadToken };++module.exports = {++ signedDownloadUrl, verifyDownload, signedFileUrl, verifyFile,++ uploadToken, uploadTtl, checkUploadToken, verifyUploadToken, multipartToken, verifyMultipartToken,++};@@@ -76,6 +76,14 @@ function domainMetrics(registry, { db, recorder, events }) {return Number.isFinite(t) ? Math.floor(t / 1000) : null; // no run yet: no series},});++ registry.gauge({++ name: 'media_jobs', help: 'Media jobs by type and status (proposed = waiting for the owner; see scripts/media-jobs.js)', labelNames: ['type', 'status'],++ collect: () => db.all('SELECT job_type, status, COUNT(*) AS n FROM media_jobs GROUP BY job_type, status').map((r) => ({ labels: { type: r.job_type, status: r.status }, value: r.n })),++ });++ registry.gauge({++ name: 'media_uploads_open', help: 'Multipart upload sessions still open (active or completing)',++ collect: () => db.get("SELECT COUNT(*) AS n FROM media_uploads WHERE status IN ('active', 'completing')").n,++ });registry.gauge({name: 'media_events_outbox', help: 'Events outbox rows waiting to reach OpenVibe.Events, and rows Events rejected', labelNames: ['status'],collect: () => {@@@ -4,7 +4,11 @@* POST /:kind/:id (upload or generate) → { url }* kind = 'vod' | 'clip' → multipart 'thumbnail' (or JSON { image: base64 })* uploads a custom image; with no image, the-- * thumbnail is (re)generated from the media file.++ * thumbnail is (re)generated from the media file by a++ * thumbnail.regenerate job (server/jobs): the route queues++ * it (joining one already queued or running for the same++ * item), runs it at once and answers { url } as before.++ * ?async=1 answers 202 { job } without waiting.* kind = 'live'|'stream' → upload only; stored under a stable per-app/id* filename so the URL survives refreshes.*@@@ -72,16 +76,29 @@ router.post('/:kind/:id', tenantAuth({ allowUser: true }), upload.single('thumbnreturn res.json({ url });}-- // Generate from the media file (local or presigned remote).-- const vodStorage = require('../vod/vod-storage');-- const source = await vodStorage.resolveMediaSource(row);-- if (!source) return res.status(404).json({ error: 'Media file unavailable' });-- const url = kind === 'vod'-- ? await thumbService.generateVodThumbnail(numId, source.value)-- : await thumbService.generateClipThumbnail(numId, source.value);-- if (!url) return res.status(500).json({ error: 'Failed to generate thumbnail' });-- res.json({ url });++ // Generate from the media file (local or presigned remote) as a thumbnail.regenerate job.++ const queue = require('../jobs/queue');++ const worker = require('../jobs/worker');++ const r = queue.enqueue({++ appId: req.appId, type: 'thumbnail.regenerate', objectId: row.object_id || null, params: { kind, id: numId },++ dedupeActive: true, maxAttempts: 1, createdBy: `app:${req.appId}${req.authType === 'user' ? `:user:${req.userId}` : ''}`,++ ownerUserId: req.authType === 'user' ? req.userId : null,++ });++ if (['1', 'true'].includes(String(req.query.async || ''))) {++ worker.kick();++ return res.status(202).json({ job: queue.jobPublic(r.job) });++ }++ if (r.created || r.job.status === 'queued') worker.runNow(r.job.id).catch(() => {}); // claims it unless someone already has++ const done = await queue.waitFor(r.job.id, 90 * 1000);++ if (!done || done.status === 'queued' || done.status === 'running') return res.status(202).json({ job: queue.jobPublic(done || r.job) });++ if (done.status === 'succeeded') return res.json({ url: queue.parseJson(done.result, {}).url, job_id: done.id });++ if (done.error_code === 'media_unavailable') return res.status(404).json({ error: 'Media file unavailable', job_id: done.id });++ res.status(500).json({ error: 'Failed to generate thumbnail', job_id: done.id });} catch (err) {++ if (err && err.code === 'media.job.too_many') {++ res.set('Retry-After', String(err.retryAfterS || 60));++ return res.status(429).json({ error: err.message });++ }console.error('[Thumbnails] Error:', err.message);res.status(500).json({ error: 'Failed to process thumbnail' });}@@@ -164,6 +164,10 @@ function bucketFor(name) {return (PROVIDER_ENV[name] && PROVIDER_ENV[name].bucket) || null;}++function endpointFor(name) {++ return (PROVIDER_ENV[name] && PROVIDER_ENV[name].endpoint) || null;++}++// Object model (Wave 4): holds freeze an object's placement, and every tier move// re-projects the VOD's media_locations. Lazy — the model requires this module.const objects = () => require('../objects/model');@@@ -1265,6 +1269,7 @@ module.exports = {isRemote,providerConfigured,bucketFor,++ endpointFor,headObject,keyForVod,localPathForVod,nnew file mode 100644@@@ -0,0 +1,226 @@++'use strict';++// The size-invariant validator job (invariant.scan) and the jobs it proposes. The validator lists public++// playback objects over MEDIA_PUBLIC_OBJECT_MAX_MB and PROPOSES object.split (duration known) or++// object.remux (duration unknown). Proposals never run by themselves: the worker leaves them alone until++// their owner approves; a rejected proposal stays rejected; a proposal whose object stopped violating is++// withdrawn; --dry-run writes nothing. Then, with ffmpeg, the approved split and a remux make private++// derived objects and leave the source untouched, and a retry resumes after the checkpointed parts.++const assert = require('assert');++const fs = require('fs');++const os = require('os');++const path = require('path');++const http = require('http');++const { spawnSync } = require('child_process');++++const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-jobinv-'));++const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++process.env.DB_PATH = path.join(tmp, 'media.db');++process.env.VOD_PATH = dir('vods');++process.env.CLIPS_PATH = dir('clips');++process.env.FILES_PATH = dir('files');++process.env.THUMBNAILS_PATH = dir('thumbnails');++process.env.PASTES_PATH = dir('pastes');++process.env.OBJECTS_PATH = dir('objects');++process.env.MEDIA_JOBS_POLL_MS = '50';++process.env.MEDIA_INVARIANT_SCAN_HOURS = '0';++process.env.MEDIA_UPLOAD_MIN_FREE_MB = '0';++++const MB = 1024 * 1024;++const stub = http.createServer((req, res) => {++ req.resume();++ req.on('end', () => {++ res.setHeader('Content-Type', 'application/json');++ if (req.url === '/oauth/token') return res.end(JSON.stringify({ access_token: 'tok', token_type: 'Bearer', expires_in: 300 }));++ res.statusCode = 404; res.end('{}');++ });++});++const sleep = (ms) => new Promise((r) => setTimeout(r, ms));++const waitFor = async (fn, ms = 20000) => {++ const end = Date.now() + ms;++ while (Date.now() < end) { if (await fn()) return true; await sleep(25); }++ return false;++};++const hasFfmpeg = spawnSync('ffmpeg', ['-version']).status === 0;++++(async () => {++ await new Promise((r) => stub.listen(0, '127.0.0.1', r));++ const base = `http://127.0.0.1:${stub.address().port}`;++ const db = require('../server/db/database');++ const conn = db.getDb();++ db.upsertApp({ app_id: 'live', api_key: 'live-key' });++ const events = require('../server/events');++ events.init({ eventsUrl: base, clientSecret: 's', networkUrl: base, intervalMs: 60000 });++ const config = require('../server/config');++ const model = require('../server/objects/model');++ const queue = require('../server/jobs/queue');++ const worker = require('../server/jobs/worker');++ const scan = require('../server/jobs/invariant-scan');++ const outbox = () => conn.prepare('SELECT envelope FROM event_outbox ORDER BY id').all().map((r) => JSON.parse(r.envelope));++ const jobEvents = (id) => outbox().filter((e) => e.subject.type === 'job' && e.subject.id === id).map((e) => e.event_type);++++ const express = require('express');++ const app = express();++ app.use(express.json());++ app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));++ const server = http.createServer(app);++ await new Promise((r) => server.listen(0, '127.0.0.1', r));++ const api = `http://127.0.0.1:${server.address().port}`;++ const call = async (method, p, { body, headers = {} } = {}) => {++ const h = { authorization: 'Bearer live-key', ...headers };++ if (body !== undefined) h['content-type'] = 'application/json';++ const res = await fetch(api + p, { method, headers: h, body: body === undefined ? undefined : JSON.stringify(body) });++ return { status: res.status, body: await res.json().catch(() => null) };++ };++ const mk = (o) => model.createObject({ app_id: 'live', owner_user_id: 5, lifecycle_status: 'ready', mime_type: 'video/webm', ...o });++++ // ── A. The validator proposes, never runs ──++ const bigVod = mk({ kind: 'vod', visibility: 'public', size_bytes: 1536 * MB, metadata: { duration_seconds: 3600 } });++ const noDur = mk({ kind: 'vod', visibility: 'unlisted', size_bytes: 800 * MB, metadata: {} });++ mk({ kind: 'vod', visibility: 'private', size_bytes: 2048 * MB, metadata: { duration_seconds: 10 } });++ mk({ kind: 'clip', visibility: 'public', size_bytes: 100 * MB });++ mk({ kind: 'clip', visibility: 'public', size_bytes: 400 * MB, metadata: { duration_seconds: 60 } });++++ const before = db.get('SELECT COUNT(*) AS n FROM media_jobs').n;++ const dry = scan.scanTenant('live', { dryRun: true });++ assert.deepStrictEqual([dry.violations, dry.proposed, dry.by_type], [2, 2, { 'object.split': 1, 'object.remux': 1 }]);++ assert.strictEqual(db.get('SELECT COUNT(*) AS n FROM media_jobs').n, before, '--dry-run proposes nothing');++ assert.strictEqual(db.get('SELECT COUNT(*) AS n FROM media_invariant_violations').n, 0, '--dry-run records nothing');++++ const s1 = queue.enqueue({ appId: 'live', type: 'invariant.scan', createdBy: 'test' });++ const ran = await worker.runNow(s1.job.id);++ assert.strictEqual(ran.status, 'succeeded', ran.error);++ const r1 = queue.jobPublic(ran).result;++ assert.deepStrictEqual([r1.violations, r1.proposed, r1.already_proposed, r1.counts.warn], [2, 2, 0, 1]);++ const proposals = db.all("SELECT * FROM media_jobs WHERE status = 'proposed' ORDER BY id").map(queue.jobPublic);++ const split = proposals.find((p) => p.object_id === bigVod);++ const remux = proposals.find((p) => p.object_id === noDur);++ assert.strictEqual(split.type, 'object.split');++ assert.deepStrictEqual([split.params.parts, split.params.segment_seconds, split.params.max_bytes], [6, 600, 500 * MB], '1.5 GB in parts of about the 256 MB target');++ assert.strictEqual(remux.type, 'object.remux', 'no duration: remux first');++ assert.strictEqual(split.idempotency_key, `invariant:${bigVod}:object.split`);++ assert.deepStrictEqual(jobEvents(split.id), ['media.job.proposed']);++ assert.strictEqual(outbox().find((e) => e.subject.id === split.id).priority, 'important', 'a proposal waiting for its owner is important');++++ worker.start();++ for (let i = 0; i < 5; i++) { worker.kick(); await sleep(60); }++ assert.deepStrictEqual([queue.get(split.id).status, queue.get(remux.id).status, queue.get(split.id).attempts], ['proposed', 'proposed', 0], 'the worker never runs a proposal');++++ const again = scan.scanTenant('live');++ assert.deepStrictEqual([again.proposed, again.already_proposed], [0, 2], 'a later scan proposes nothing twice');++ let r = await call('POST', `/api/v2/live/jobs/${remux.id}/cancel`);++ assert.deepStrictEqual([r.status, r.body.job.status, r.body.job.decided_by], [200, 'cancelled', 'app:live'], 'the owner rejects a proposal');++ scan.scanTenant('live');++ assert.strictEqual(queue.get(remux.id).status, 'cancelled', 'a rejected proposal stays rejected');++ assert.strictEqual(db.get('SELECT COUNT(*) AS n FROM media_jobs WHERE object_id = ?', [noDur]).n, 1, 'and is not proposed again');++ model.updateObject(bigVod, { visibility: 'private' });++ const w = scan.scanTenant('live');++ assert.strictEqual(w.withdrawn, 1);++ assert.deepStrictEqual([queue.get(split.id).status, queue.get(split.id).decided_by], ['cancelled', 'system:invariant.scan'], 'a proposal whose object stopped violating is withdrawn');++++ config.jobs.invariantScanHours = 24;++ model.updateObject(bigVod, { visibility: 'public' });++ const scheduled = worker.scheduleInvariantScans(Date.now());++ assert.strictEqual(scheduled, 1, 'one scheduled scan per tenant with public playback objects');++ assert.strictEqual(worker.scheduleInvariantScans(Date.now()), 0, 'once per period');++ config.jobs.invariantScanHours = 0;++ assert.ok(await waitFor(() => db.get("SELECT status FROM media_jobs WHERE job_type = 'invariant.scan' AND created_by = 'system:schedule'").status === 'succeeded'));++ console.log('✅ invariant.scan proposes split/remux for public objects over the limit; nothing runs until the owner decides');++++ // ── B. Approved and explicit split/remux jobs (ffmpeg) ──++ r = await call('POST', '/api/v2/live/jobs', { body: { type: 'object.split', object_id: model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: 'ready', size_bytes: 1 }), params: { parts: 2 } } });++ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.job.invalid'], 'only media objects are split');++ r = await call('POST', '/api/v2/live/jobs', { body: { type: 'object.split', object_id: bigVod, params: { parts: 1 } } });++ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.job.invalid'], 'parts 2-1000');++++ if (!hasFfmpeg) {++ console.log('⚠️ ffmpeg not found: split/remux execution skipped');++ } else {++ const src = path.join(process.env.VOD_PATH, 'rec-10.mp4');++ const mk6 = spawnSync('ffmpeg', ['-v', 'error', '-y', '-f', 'lavfi', '-i', 'testsrc=duration=6:size=160x120:rate=10', '-f', 'lavfi', '-i', 'sine=frequency=440:duration=6',++ '-c:v', 'mpeg4', '-g', '10', '-c:a', 'aac', '-shortest', src]);++ assert.strictEqual(mk6.status, 0, String(mk6.stderr));++ const size = fs.statSync(src).size;++ conn.prepare("INSERT INTO vods (id, app_id, user_id, title, file_path, file_size, is_public, duration_seconds) VALUES (10, 'live', 5, 'Six seconds', ?, ?, 1, 6)").run(src, size);++ const vodObj = model.sync('vod', 10).id;++ const srcBefore = model.getObject(vodObj);++ const locsBefore = JSON.stringify(model.listLocations(vodObj));++++ // The validator proposes a split once the policy says the object is too large; the owner approves it.++ config.objects.publicMaxMb = 0;++ config.objects.publicTargetMb = 0;++ scan.scanTenant('live');++ const prop = queue.jobPublic(db.get("SELECT * FROM media_jobs WHERE object_id = ? AND status = 'proposed'", [vodObj]));++ assert.deepStrictEqual([prop.type, prop.params.segment_seconds], ['object.split', 1]);++ r = await call('POST', `/api/v2/live/jobs/${prop.id}/approve`);++ assert.deepStrictEqual([r.status, r.body.job.status, r.body.job.decided_by], [200, 'queued', 'app:live']);++ assert.ok(await waitFor(() => ['succeeded', 'failed'].includes(queue.get(prop.id).status)), 'the approved split runs');++ const done = queue.jobPublic(queue.get(prop.id));++ assert.strictEqual(done.status, 'succeeded', done.error);++ assert.deepStrictEqual(jobEvents(prop.id), ['media.job.proposed', 'media.job.queued', 'media.job.started', 'media.job.succeeded']);++ assert.strictEqual(done.result.parts.length, 6, '6 s in 1 s parts');++ let total = 0;++ for (const p of done.result.parts) {++ const o = model.getObject(p.object_id);++ assert.deepStrictEqual([o.visibility, o.lifecycle_status, o.kind, o.app_id, o.owner_user_id, o.legacy_ref, o.mime_type], ['private', 'ready', 'vod', 'live', 5, null, 'video/mp4']);++ const loc = model.listLocations(o.id)[0];++ assert.deepStrictEqual([loc.provider, loc.state, loc.size_bytes], ['local', 'present', o.size_bytes]);++ assert.ok(fs.existsSync(loc.key) && loc.key.startsWith(path.join(process.env.OBJECTS_PATH, 'live')));++ const rel = db.get("SELECT * FROM media_relationships WHERE from_object_id = ? AND relation = 'derived_from'", [o.id]);++ assert.strictEqual(rel.to_object_id, vodObj);++ assert.deepStrictEqual([JSON.parse(rel.metadata).job_id, JSON.parse(rel.metadata).part], [prop.id, p.part]);++ total += p.duration_seconds;++ }++ assert.ok(Math.abs(total - 6) < 0.5, 'the parts cover the source');++ const srcAfter = model.getObject(vodObj);++ assert.deepStrictEqual([srcAfter.visibility, srcAfter.size_bytes, srcAfter.lifecycle_status, srcAfter.content_hash], [srcBefore.visibility, srcBefore.size_bytes, 'ready', srcBefore.content_hash], 'the source is unchanged');++ assert.strictEqual(JSON.stringify(model.listLocations(vodObj)), locsBefore);++ assert.strictEqual(fs.statSync(src).size, size, 'its bytes too');++ assert.ok(!fs.existsSync(path.join(process.env.OBJECTS_PATH, '.jobs', prop.id)), 'the work directory is removed');++ config.objects.publicMaxMb = 500;++ config.objects.publicTargetMb = 256;++++ // A retry resumes after the parts its checkpoint names.++ const resumed = queue.enqueue({ appId: 'live', type: 'object.split', objectId: vodObj, params: { parts: 3 }, createdBy: 'test' });++ queue.saveCheckpoint(resumed.job.id, { parts: [{ part: 1, object_id: 'med_done_before', start_seconds: 0, duration_seconds: 2, size_bytes: 10 }] });++ assert.ok(await waitFor(() => queue.get(resumed.job.id).status === 'succeeded'));++ const rr = queue.jobPublic(queue.get(resumed.job.id)).result;++ assert.deepStrictEqual(rr.parts.map((p) => p.part), [1, 2, 3]);++ assert.strictEqual(rr.parts[0].object_id, 'med_done_before', 'part 1 was not cut again');++ assert.strictEqual(db.get("SELECT COUNT(*) AS n FROM media_relationships WHERE relation = 'derived_from' AND json_extract(metadata, '$.job_id') = ?", [resumed.job.id]).n, 2);++++ // A remux: one private object, the source's remux variant.++ r = await call('POST', '/api/v2/live/jobs', { body: { type: 'object.remux', object_id: `legacy:live:vod:10` }, headers: { 'idempotency-key': 'remux-10' } });++ assert.strictEqual(r.status, 202);++ assert.ok(await waitFor(() => queue.get(r.body.job.id).status === 'succeeded'));++ const rm = queue.jobPublic(queue.get(r.body.job.id)).result;++ assert.deepStrictEqual([model.getObject(rm.object_id).visibility, Math.round(rm.duration_seconds), model.getVariant(vodObj, 'remux').derived_object_id], ['private', 6, rm.object_id]);++ console.log('✅ an approved split and a remux make private derived objects; the source is untouched; retries resume');++ }++++ // ── C. The operator script: scan (dry run by default), approve, cancel; events go to the service's outbox ──++ worker.stop();++ const script = (...a) => spawnSync(process.execPath, [path.join(__dirname, '../scripts/media-jobs.js'), ...a], { env: process.env, encoding: 'utf8' });++ const jobsBefore = db.get('SELECT COUNT(*) AS n FROM media_jobs').n;++ let sc = script('scan', '--app', 'live', '--json');++ assert.strictEqual(sc.status, 0, sc.stderr);++ assert.strictEqual(JSON.parse(sc.stdout).live.dry_run, true);++ assert.strictEqual(db.get('SELECT COUNT(*) AS n FROM media_jobs').n, jobsBefore, 'scan is a dry run unless --apply');++ const manual = queue.enqueue({ appId: 'live', type: 'object.remux', objectId: bigVod, status: 'proposed', createdBy: 'test' });++ sc = script('approve', manual.job.id, '--by', 'owner');++ assert.strictEqual(sc.status, 0, sc.stderr);++ assert.deepStrictEqual([queue.get(manual.job.id).status, queue.get(manual.job.id).decided_by], ['queued', 'owner']);++ assert.ok(jobEvents(manual.job.id).includes('media.job.queued'), "the script's change queued its event in the outbox");++ sc = script('cancel', manual.job.id, '--by', 'owner');++ assert.strictEqual(sc.status, 0, sc.stderr);++ assert.strictEqual(queue.get(manual.job.id).status, 'cancelled');++ assert.strictEqual(script('approve', 'mjob_nope').status, 1);++ sc = script('list', '--status', 'cancelled', '--json');++ assert.ok(JSON.parse(sc.stdout).jobs.some((j) => j.id === manual.job.id));++ console.log('✅ scripts/media-jobs.js: scan dry-run, approve and cancel with their events');++++ events._reset();++ server.close();++ stub.close();++ console.log('\njobs-invariant: all checks passed');++ process.exit(0);++})().catch((err) => { console.error(err); process.exit(1); });nnew file mode 100644@@@ -0,0 +1,257 @@++'use strict';++// Media job system (server/jobs): the schema-only media_jobs table is rebuilt; every state change++// commits with its media.job.* outbox row (and rolls back with it); idempotency keys; retries with++// backoff, permanent failures, checkpoints; cancellation by the owner (queued at once, running through++// the abort signal) and not by anyone else; interrupted jobs are requeued at start; the v1 thumbnail++// route runs as a thumbnail.regenerate job and answers as before.++const assert = require('assert');++const fs = require('fs');++const os = require('os');++const path = require('path');++const http = require('http');++++const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-jobs-'));++const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++process.env.DB_PATH = path.join(tmp, 'media.db');++process.env.VOD_PATH = dir('vods');++process.env.CLIPS_PATH = dir('clips');++process.env.FILES_PATH = dir('files');++process.env.THUMBNAILS_PATH = dir('thumbnails');++process.env.PASTES_PATH = dir('pastes');++process.env.OBJECTS_PATH = dir('objects');++process.env.MEDIA_JOBS_POLL_MS = '50';++process.env.MEDIA_JOBS_LEASE_S = '30';++process.env.MEDIA_INVARIANT_SCAN_HOURS = '0';++++// ── 1. A database from before the worker: the schema-only table is rebuilt, rows kept ──++{++ const Database = require('better-sqlite3');++ const old = new Database(process.env.DB_PATH);++ old.exec(`CREATE TABLE media_jobs (id INTEGER PRIMARY KEY AUTOINCREMENT, object_id TEXT, job_type TEXT NOT NULL,++ status TEXT NOT NULL DEFAULT 'queued' CHECK(status IN ('queued', 'running', 'done', 'failed', 'cancelled')),++ attempts INTEGER NOT NULL DEFAULT 0, checkpoint TEXT, error TEXT, created_at DATETIME DEFAULT CURRENT_TIMESTAMP,++ updated_at DATETIME DEFAULT CURRENT_TIMESTAMP);++ CREATE INDEX idx_media_jobs_status ON media_jobs(status, job_type);++ INSERT INTO media_jobs (object_id, job_type, status, attempts) VALUES (NULL, 'legacy.thing', 'done', 1);`);++ old.close();++}++++const hooks = [];++const stub = http.createServer((req, res) => {++ let body = '';++ req.on('data', (c) => { body += c; });++ req.on('end', () => {++ res.setHeader('Content-Type', 'application/json');++ if (req.url === '/oauth/token') return res.end(JSON.stringify({ access_token: 'tok', token_type: 'Bearer', expires_in: 300 }));++ if (req.url === '/hook') { hooks.push(JSON.parse(body)); return res.end('{}'); }++ res.statusCode = 404; res.end('{}');++ });++});++++const sleep = (ms) => new Promise((r) => setTimeout(r, ms));++const waitFor = async (fn, ms = 5000) => {++ const end = Date.now() + ms;++ while (Date.now() < end) { if (await fn()) return true; await sleep(20); }++ return false;++};++++(async () => {++ await new Promise((r) => stub.listen(0, '127.0.0.1', r));++ const base = `http://127.0.0.1:${stub.address().port}`;++ const db = require('../server/db/database');++ const raw = db.getDb();++++ const cols = raw.prepare('PRAGMA table_info(media_jobs)').all().map((c) => c.name);++ assert.ok(cols.includes('app_id') && cols.includes('idempotency_key') && cols.includes('lease_until'), 'media_jobs rebuilt into the worker shape');++ const legacy = raw.prepare("SELECT * FROM media_jobs WHERE id = 'mjob_legacy_1'").get();++ assert.deepStrictEqual([legacy.job_type, legacy.status, legacy.app_id], ['legacy.thing', 'succeeded', 'unknown'], 'the old row is kept, done -> succeeded');++ assert.ok(raw.prepare("SELECT 1 FROM sqlite_master WHERE name = 'idx_media_jobs_idem'").get(), 'indexes on the new columns exist');++ db.close();++ db.getDb(); // a second open is a no-op++ assert.ok(db.get("SELECT 1 FROM media_jobs WHERE id = 'mjob_legacy_1'"));++ console.log('✅ the schema-only media_jobs table is rebuilt (rows kept); reopening is idempotent');++++ db.upsertApp({ app_id: 'live', api_key: 'live-key', webhook_url: `${base}/hook`, webhook_secret: 's' });++ db.upsertApp({ app_id: 'games', api_key: 'games-key' });++ const events = require('../server/events');++ assert.ok(events.init({ eventsUrl: base, clientSecret: 's', networkUrl: base, intervalMs: 60000 }));++ const conn = db.getDb();++ const outbox = () => conn.prepare('SELECT envelope FROM event_outbox ORDER BY id').all().map((r) => JSON.parse(r.envelope));++ const jobEvents = (id) => outbox().filter((e) => e.subject.type === 'job' && e.subject.id === id).map((e) => e.event_type);++ const failOutbox = (on) => conn.exec(on++ ? "CREATE TEMP TRIGGER outbox_boom BEFORE INSERT ON event_outbox BEGIN SELECT RAISE(ABORT, 'outbox insert failed'); END"++ : 'DROP TRIGGER IF EXISTS temp.outbox_boom');++++ const queue = require('../server/jobs/queue');++ const worker = require('../server/jobs/worker');++++ // Test job types.++ let flakyCalls = 0;++ queue.register('test.flaky', {++ lane: 'light', maxAttempts: 3, backoffS: () => 0,++ run: async (job, ctx) => {++ flakyCalls++;++ if (!ctx.checkpoint) { ctx.saveCheckpoint({ step: 1 }); throw new Error('transient'); }++ return { resumed_from: ctx.checkpoint.step };++ },++ });++ queue.register('test.permanent', { lane: 'light', run: async () => { throw new queue.JobError('bad_input', 'will never work', { permanent: true }); } });++ queue.register('test.always', { lane: 'light', maxAttempts: 2, backoffS: () => 0, run: async () => { throw new Error('nope'); } });++ let release = null;++ queue.register('test.slow', {++ lane: 'heavy',++ run: (job, ctx) => new Promise((resolve, reject) => {++ release = resolve;++ ctx.signal.addEventListener('abort', () => reject(ctx.signal.reason), { once: true });++ }),++ });++++ // ── 2. A state change and its event commit together ──++ const a = queue.enqueue({ appId: 'live', type: 'test.flaky', params: { x: 1 }, createdBy: 'test' });++ assert.ok(a.created && /^mjob_[0-9A-HJKMNP-TV-Z]{26}$/.test(a.job.id));++ assert.deepStrictEqual(jobEvents(a.job.id), ['media.job.queued']);++ const env = outbox().find((e) => e.subject.id === a.job.id);++ assert.deepStrictEqual([env.source, env.priority, env.visibility, env.payload.type, env.payload.status, env.payload.app_id], ['media', 'low', 'internal', 'test.flaky', 'queued', 'live']);++ const q2 = queue.enqueue({ appId: 'live', type: 'test.permanent' });++ failOutbox(true);++ assert.throws(() => queue.enqueue({ appId: 'live', type: 'test.flaky', params: { x: 2 } }), /outbox insert failed/);++ assert.strictEqual(db.get("SELECT COUNT(*) AS n FROM media_jobs WHERE params = '{\"x\":2}'").n, 0, 'no job without its event');++ assert.throws(() => queue.cancel(q2.job.id, { by: 'x' }), /outbox insert failed/);++ failOutbox(false);++ assert.strictEqual(queue.get(q2.job.id).status, 'queued', 'a cancel whose event cannot be written changes nothing');++ assert.ok(hooks.length === 0, 'job events go to Events only, not the app webhook');++ console.log('✅ job state changes commit with their media.job.* outbox row, or not at all');++++ // ── 3. Idempotency keys ──++ const k1 = queue.enqueue({ appId: 'live', type: 'test.permanent', params: { n: 1 }, idempotencyKey: 'key-1' });++ const k2 = queue.enqueue({ appId: 'live', type: 'test.permanent', params: { n: 1 }, idempotencyKey: 'key-1' });++ assert.ok(k1.created && k2.replayed && k1.job.id === k2.job.id);++ assert.throws(() => queue.enqueue({ appId: 'live', type: 'test.permanent', params: { n: 2 }, idempotencyKey: 'key-1' }), (e) => e.code === 'media.job.idempotency_conflict' && e.status === 409);++ assert.ok(queue.enqueue({ appId: 'games', type: 'test.permanent', params: { n: 1 }, idempotencyKey: 'key-1' }).created, 'keys are per tenant');++ const d1 = queue.enqueue({ appId: 'live', type: 'test.permanent', params: { same: true }, dedupeActive: true });++ const d2 = queue.enqueue({ appId: 'live', type: 'test.permanent', params: { same: true }, dedupeActive: true });++ assert.ok(d2.deduped && d2.job.id === d1.job.id, 'dedupeActive joins an identical queued job');++ for (const id of [q2.job.id, k1.job.id, d1.job.id]) queue.cancel(id, { by: 'test' });++ db.run("UPDATE media_jobs SET status = 'cancelled' WHERE app_id = 'games'");++ console.log('✅ idempotency keys: replay, per-tenant, conflict on a different request; active-job dedupe');++++ // ── 4. Worker: retry with backoff + checkpoint resume, permanent failure, attempts exhausted ──++ assert.ok(worker.start());++ const perm = queue.enqueue({ appId: 'live', type: 'test.permanent' });++ const always = queue.enqueue({ appId: 'live', type: 'test.always' });++ worker.kick();++ assert.ok(await waitFor(() => queue.get(a.job.id).status === 'succeeded'), 'the flaky job succeeds on its retry');++ assert.strictEqual(flakyCalls, 2);++ assert.deepStrictEqual(queue.jobPublic(queue.get(a.job.id)).result, { resumed_from: 1 }, 'the retry resumed from the checkpoint');++ assert.deepStrictEqual(jobEvents(a.job.id), ['media.job.queued', 'media.job.started', 'media.job.retrying', 'media.job.started', 'media.job.succeeded']);++ assert.ok(await waitFor(() => queue.get(perm.job.id).status === 'failed'));++ assert.deepStrictEqual([queue.get(perm.job.id).attempts, queue.get(perm.job.id).error_code], [1, 'bad_input'], 'a permanent failure is not retried');++ assert.ok(await waitFor(() => queue.get(always.job.id).status === 'failed'));++ assert.strictEqual(queue.get(always.job.id).attempts, 2, 'retried until max_attempts');++ assert.deepStrictEqual(jobEvents(always.job.id), ['media.job.queued', 'media.job.started', 'media.job.retrying', 'media.job.started', 'media.job.failed']);++ const failedEv = outbox().filter((e) => e.subject.id === always.job.id).pop();++ assert.deepStrictEqual([failedEv.priority, failedEv.payload.error], ['important', 'nope']);++ console.log('✅ worker: retries with backoff resume from the checkpoint; permanent failures stop; attempts are capped');++++ // ── 5. Cancellation by the owner ──++ const slow = queue.enqueue({ appId: 'live', type: 'test.slow', createdBy: 'app:live:user:7', ownerUserId: 7 });++ worker.kick();++ assert.ok(await waitFor(() => queue.get(slow.job.id).status === 'running'));++ const express = require('express');++ const app = express();++ app.use(express.json());++ app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));++ app.use('/api/v1/:app/thumbnails', require('../server/thumbnails/routes'));++ const server = http.createServer(app);++ await new Promise((r) => server.listen(0, '127.0.0.1', r));++ const api = `http://127.0.0.1:${server.address().port}`;++ const call = async (method, p, { key = 'live-key', body, headers = {} } = {}) => {++ const h = { authorization: `Bearer ${key}`, ...headers };++ if (body !== undefined) h['content-type'] = 'application/json';++ const res = await fetch(api + p, { method, headers: h, body: body === undefined ? undefined : JSON.stringify(body) });++ const text = await res.text();++ let json = null; try { json = JSON.parse(text); } catch { /* */ }++ return { status: res.status, body: json, headers: res.headers };++ };++ let r = await call('POST', `/api/v2/live/jobs/${slow.job.id}/cancel`, { headers: { 'x-ov-user-id': '8' } });++ assert.strictEqual(r.status, 404, 'another user cannot see or cancel it');++ r = await call('POST', `/api/v2/games/jobs/${slow.job.id}/cancel`, { key: 'games-key' });++ assert.strictEqual(r.status, 404, 'another tenant cannot either');++ r = await call('POST', `/api/v2/live/jobs/${slow.job.id}/cancel`, { headers: { 'x-ov-user-id': '7' } });++ assert.deepStrictEqual([r.status, r.body.job.cancel_requested], [202, true], 'a running job stops at its next check');++ assert.ok(await waitFor(() => queue.get(slow.job.id).status === 'cancelled'), 'the abort signal stopped it');++ assert.deepStrictEqual(jobEvents(slow.job.id), ['media.job.queued', 'media.job.started', 'media.job.cancelled']);++ r = await call('DELETE', `/api/v2/live/jobs/${slow.job.id}`);++ assert.deepStrictEqual([r.status, r.body.code], [409, 'media.job.finished']);++ const waiting = queue.enqueue({ appId: 'live', type: 'test.permanent', runAfterS: 3600 });++ r = await call('DELETE', `/api/v2/live/jobs/${waiting.job.id}`);++ assert.deepStrictEqual([r.status, r.body.job.status], [200, 'cancelled'], 'a queued job is cancelled at once');++ console.log('✅ cancellation: owner only; queued at once, running through the abort signal; finished = 409');++++ // ── 6. Interrupted jobs are requeued at start ──++ worker.stop();++ const orphan = queue.enqueue({ appId: 'live', type: 'test.permanent', params: { orphan: true } });++ db.run("UPDATE media_jobs SET status = 'running', attempts = 1, lease_until = datetime('now', '+1 hour') WHERE id = ?", [orphan.job.id]);++ assert.strictEqual(queue.recoverInterrupted({ all: true }), 1);++ const back = queue.get(orphan.job.id);++ assert.deepStrictEqual([back.status, back.error_code], ['queued', 'interrupted']);++ assert.deepStrictEqual(jobEvents(orphan.job.id).slice(-1), ['media.job.retrying']);++ queue.cancel(orphan.job.id, { by: 'test' });++ console.log('✅ a job left running by a dead process is requeued (or failed when out of attempts)');++++ // ── 7. The jobs API ──++ r = await call('POST', '/api/v2/live/jobs', { body: { type: 'nope' } });++ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.job.unknown_type']);++ r = await call('POST', '/api/v2/live/jobs', { body: { type: 'object.split' } });++ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.job.invalid'], 'object.split needs an object');++ r = await call('POST', '/api/v2/live/jobs', { body: { type: 'invariant.scan' }, headers: { 'idempotency-key': 'scan-1' } });++ assert.strictEqual(r.status, 202);++ const scanId = r.body.job.id;++ r = await call('POST', '/api/v2/live/jobs', { body: { type: 'invariant.scan' }, headers: { 'idempotency-key': 'scan-1' } });++ assert.deepStrictEqual([r.status, r.body.job.id, r.headers.get('idempotent-replayed')], [200, scanId, 'true']);++ r = await call('POST', `/api/v2/live/jobs/${scanId}/approve`);++ assert.deepStrictEqual([r.status, r.body.code], [409, 'media.job.not_proposed']);++ r = await call('GET', `/api/v2/live/jobs?type=invariant.scan`);++ assert.deepStrictEqual(r.body.jobs.map((j) => j.id), [scanId]);++ r = await call('GET', `/api/v2/games/jobs/${scanId}`, { key: 'games-key' });++ assert.strictEqual(r.status, 404, 'tenants are isolated');++ r = await call('GET', `/api/v2/live/jobs?status=bogus`);++ assert.strictEqual(r.status, 400);++ queue.cancel(scanId, { by: 'test' });++ console.log('✅ jobs API: create (Idempotency-Key replay), get, list, approve only proposals, tenant isolation');++++ // ── 8. The v1 thumbnail route runs as a thumbnail.regenerate job ──++ const thumbs = require('../server/thumbnails/thumbnail-service');++ let gens = 0;++ thumbs.generateVodThumbnail = async (id) => { gens++; await sleep(150); db.run('UPDATE vods SET thumbnail_url = ? WHERE id = ?', [`/t/vod-${id}-1.jpg`, id]); return `/t/vod-${id}-1.jpg`; };++ fs.writeFileSync(path.join(process.env.VOD_PATH, 'rec-1.webm'), 'not really a video');++ conn.prepare("INSERT INTO vods (id, app_id, user_id, title, file_path, is_public, duration_seconds) VALUES (1, 'live', 5, 'A', ?, 1, 60)").run(path.join(process.env.VOD_PATH, 'rec-1.webm'));++ conn.prepare("INSERT INTO vods (id, app_id, user_id, title, file_path, is_public, duration_seconds) VALUES (2, 'live', 5, 'B', ?, 1, 60)").run(path.join(process.env.VOD_PATH, 'gone.webm'));++ require('../server/objects/model').sync('vod', 1);++ const [t1, t2] = await Promise.all([call('POST', '/api/v1/live/thumbnails/vod/1'), call('POST', '/api/v1/live/thumbnails/vod/1')]);++ assert.deepStrictEqual([t1.status, t1.body.url, t2.status, t2.body.url], [200, '/t/vod-1-1.jpg', 200, '/t/vod-1-1.jpg'], 'the answer keeps its { url } shape');++ assert.strictEqual(t1.body.job_id, t2.body.job_id, 'two concurrent requests share one job');++ assert.strictEqual(gens, 1, 'one generation for both');++ const tj = queue.get(t1.body.job_id);++ assert.deepStrictEqual([tj.job_type, tj.status, tj.object_id], ['thumbnail.regenerate', 'succeeded', db.get('SELECT object_id FROM vods WHERE id = 1').object_id]);++ assert.deepStrictEqual(jobEvents(tj.id), ['media.job.queued', 'media.job.started', 'media.job.succeeded']);++ r = await call('POST', '/api/v1/live/thumbnails/vod/2');++ assert.deepStrictEqual([r.status, r.body.error], [404, 'Media file unavailable'], 'no media is still a 404');++ assert.strictEqual(queue.get(r.body.job_id).error_code, 'media_unavailable');++ r = await call('POST', '/api/v1/live/thumbnails/vod/1?async=1');++ assert.strictEqual(r.status, 202);++ assert.ok(await waitFor(() => false, 50) || true);++ worker.start();++ assert.ok(await waitFor(() => queue.get(r.body.job.id).status === 'succeeded'), '?async=1 queues it for the worker');++ r = await call('POST', '/api/v2/live/jobs', { body: { type: 'thumbnail.regenerate', object_id: `legacy:live:vod:1` } });++ assert.deepStrictEqual([r.status, r.body.job.params], [202, { kind: 'vod', id: 1 }], 'the v2 API takes the object and finds the row');++ assert.ok(await waitFor(() => queue.get(r.body.job.id).status === 'succeeded'));++ console.log('✅ the v1 thumbnail route runs a thumbnail.regenerate job and answers { url } as before (404 without media)');++++ worker.stop();++ events._reset();++ server.close();++ stub.close();++ console.log('\njobs: all checks passed');++ process.exit(0);++})().catch((err) => { console.error(err); process.exit(1); });nnew file mode 100644@@@ -0,0 +1,231 @@++'use strict';++// Object API v2 uploads for large files: presigned single-PUT URLs scoped to tenant, object and size++// (MEDIA_SIGNING_SECRET, expiring); multipart initiate / part / status / complete / abort, resumable++// after a dropped connection; content-type rules per kind and byte checks for inline-served types;++// tenant quota and size limits; expired sessions purged.++const assert = require('assert');++const fs = require('fs');++const os = require('os');++const path = require('path');++const http = require('http');++const crypto = require('crypto');++++const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-multipart-'));++const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++process.env.DB_PATH = path.join(tmp, 'media.db');++process.env.VOD_PATH = dir('vods');++process.env.FILES_PATH = dir('files');++process.env.OBJECTS_PATH = dir('objects');++process.env.THUMBNAILS_PATH = dir('thumbnails');++process.env.MEDIA_PUBLIC_URL = 'https://media.test';++process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';++process.env.MEDIA_OBJECT_MAX_MB = '1'; // single-part limit++process.env.MEDIA_MULTIPART_MAX_MB = '4';++process.env.MEDIA_MULTIPART_MIN_PART_MB = '0'; // small parts in tests++process.env.MEDIA_UPLOAD_MIN_FREE_MB = '0';++++const db = require('../server/db/database');++const model = require('../server/objects/model');++const multipart = require('../server/objects/multipart');++const objectRoutes = require('../server/objects/routes');++const express = require('express');++++db.upsertApp({ app_id: 'live', api_key: 'live-key' });++db.upsertApp({ app_id: 'games', api_key: 'games-key' });++db.upsertApp({ app_id: 'tiny', api_key: 'tiny-key', quota_bytes: 1024 * 1024 });++++const app = express();++app.put('/api/v2/:app/objects/:id/content', ...objectRoutes.contentHandlers);++app.put('/api/v2/:app/objects/:id/multipart/:uploadId/parts/:n', ...objectRoutes.partHandlers);++app.use(express.json());++app.use('/api/v2/:app/objects', objectRoutes);++app.use('/o', objectRoutes.publicRouter);++const server = http.createServer(app);++const sha = (b) => crypto.createHash('sha256').update(b).digest('hex');++const mac = (parts) => crypto.createHmac('sha256', 'test-signing-secret').update(parts.join('\n')).digest('base64url');++++(async () => {++ await new Promise((r) => server.listen(0, '127.0.0.1', r));++ const base = `http://127.0.0.1:${server.address().port}`;++ const local = (u) => base + String(u).replace('https://media.test', '');++ const call = async (method, p, { bearer = 'live-key', body, headers = {}, raw } = {}) => {++ const h = { ...headers };++ if (bearer) h.authorization = `Bearer ${bearer}`;++ let payload;++ if (raw !== undefined) payload = raw;++ else if (body !== undefined) { h['content-type'] = 'application/json'; payload = JSON.stringify(body); }++ const res = await fetch(p.startsWith('http') ? p : base + p, { method, headers: h, body: payload, redirect: 'manual' });++ const text = await res.text();++ let json = null; try { json = JSON.parse(text); } catch { /* not json */ }++ return { status: res.status, body: json, text, headers: res.headers };++ };++ const O = '/api/v2/live/objects';++ const now = Math.floor(Date.now() / 1000);++++ // ── 1. Presigned single PUT: scoped to tenant, object and size; expiring ──++ const hello = Buffer.from('hello world');++ let r = await call('POST', O, { body: { kind: 'file', mime_type: 'text/plain', size_bytes: hello.length, upload_ttl: 120 } });++ assert.strictEqual(r.status, 201, JSON.stringify(r.body));++ const a = r.body;++ assert.ok(/^v2\.\d+\.11\.[\w-]+$/.test(a.upload.token), 'the token names its size');++ assert.ok(Math.abs(Date.parse(a.upload.expires_at) - (Date.now() + 120000)) < 5000, 'upload_ttl sets the expiry');++ assert.deepStrictEqual([a.upload.method, a.upload.content_type, a.upload.max_bytes], ['PUT', 'text/plain', 11]);++ r = await call('POST', O, { body: { kind: 'file', mime_type: 'text/plain', size_bytes: hello.length } });++ const b = r.body;++ r = await call('PUT', local(b.upload.url).replace(b.id, a.id), { bearer: null, raw: hello });++ assert.deepStrictEqual([r.status, r.body.code], [401, 'media.upload_token.invalid'], "another object's token");++ const [, exp, , sig] = a.upload.token.split('.');++ r = await call('PUT', `${O}/${a.id}/content?token=v2.${exp}.12.${sig}`, { bearer: null, raw: Buffer.from('hello world!') });++ assert.strictEqual(r.status, 401, 'a token cannot be re-scoped to another size');++ r = await call('PUT', local(a.upload.url).replace('/api/v2/live/', '/api/v2/games/'), { bearer: null, raw: hello });++ assert.strictEqual(r.status, 404, 'nor used on another tenant');++ const past = now - 5;++ r = await call('PUT', `${O}/${a.id}/content?token=v2.${past}.11.${mac(['put2', 'live', a.id, 11, past])}`, { bearer: null, raw: hello });++ assert.strictEqual(r.status, 401, 'expired');++ const otherTenant = now + 100;++ r = await call('PUT', `${O}/${a.id}/content?token=v2.${otherTenant}.11.${mac(['put2', 'games', a.id, 11, otherTenant])}`, { bearer: null, raw: hello });++ assert.strictEqual(r.status, 401, 'a token signed for another tenant');++ r = await call('PUT', local(a.upload.url), { bearer: null, raw: hello });++ assert.strictEqual(r.status, 200, JSON.stringify(r.body));++ r = await call('POST', `${O}/${a.id}/complete?token=${encodeURIComponent(a.upload.token)}`, { bearer: null });++ assert.deepStrictEqual([r.status, r.body.lifecycle_status], [200, 'ready']);++ r = await call('POST', `${O}/${b.id}/upload-url`, { body: { ttl: 300 } });++ assert.ok(r.status === 200 && r.body.url.includes('token=v2.'), 'a fresh presigned URL');++ r = await call('POST', `${O}/${b.id}/upload-url`, { body: { ttl: 5 } });++ assert.strictEqual(r.status, 400);++ const legacyExp = now + 300;++ r = await call('PUT', `${O}/${b.id}/content?token=${legacyExp}.${crypto.createHmac('sha256', 'test-signing-secret').update(`put\n${b.id}\n${legacyExp}`).digest('base64url')}`, { bearer: null, raw: hello });++ assert.strictEqual(r.status, 200, 'tokens of the earlier form work until they expire');++ console.log('✅ presigned single-PUT URLs: scoped to tenant, object and size, expiring');++++ // ── 2. Content types ──++ r = await call('POST', O, { body: { kind: 'screenshot', mime_type: 'text/html' } });++ assert.deepStrictEqual([r.status, r.body.code], [415, 'media.object.unsupported_type']);++ r = await call('POST', O, { body: { kind: 'vod', visibility: 'private', mime_type: 'image/png' } });++ assert.strictEqual(r.status, 415, 'a vod is video or audio');++ r = await call('POST', O, { body: { kind: 'avatar', mime_type: 'image/svg+xml' } });++ assert.strictEqual(r.status, 415, 'no SVG where images are served inline');++ r = await call('POST', O, { body: { kind: 'screenshot', mime_type: 'image/png' } });++ const fake = r.body.id;++ r = await call('PUT', `${O}/${fake}/content`, { raw: Buffer.from('<html><script>alert(1)</script></html>') });++ assert.strictEqual(r.status, 200);++ r = await call('POST', `${O}/${fake}/complete`);++ assert.deepStrictEqual([r.status, r.body.code], [415, 'media.object.content_mismatch'], 'bytes that are not the declared image');++ assert.strictEqual(model.getObject(fake).lifecycle_status, 'uploading', 'the object stays uploading');++ r = await call('POST', O, { body: { kind: 'screenshot' } });++ r = await call('PUT', `${O}/${r.body.id}/content`, { raw: Buffer.from('<svg/>'), headers: { 'content-type': 'image/svg+xml' } });++ assert.deepStrictEqual([r.status, r.body.code], [415, 'media.object.unsupported_type'], 'an undeclared type comes from the PUT and is checked too');++ r = await call('POST', O, { body: { kind: 'file', mime_type: 'image/gif' } });++ const gifId = r.body.id;++ await call('PUT', `${O}/${gifId}/content`, { raw: Buffer.from('%PDF-1.4 not a gif') });++ r = await call('POST', `${O}/${gifId}/complete`);++ assert.deepStrictEqual([r.status, r.body.code], [415, 'media.object.content_mismatch'], 'inline-served types are checked for any kind');++ r = await call('POST', O, { body: { kind: 'file', mime_type: 'application/zip' } });++ const zipId = r.body.id;++ await call('PUT', `${O}/${zipId}/content`, { raw: Buffer.from('anything at all') });++ assert.strictEqual((await call('POST', `${O}/${zipId}/complete`)).status, 200, 'attachments are taken as declared');++ console.log('✅ content types: kind rules at init and PUT, byte checks at complete for inline-served types');++++ // ── 3. Multipart ──++ const PART = 512 * 1024;++ const big = crypto.randomBytes(PART * 2 + 251424);++ r = await call('POST', O, { body: { kind: 'file', mime_type: 'application/octet-stream', size_bytes: big.length } });++ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.object.too_large'], 'over the single-part limit without multipart');++ assert.match(r.body.detail, /multipart: true/);++ r = await call('POST', O, { body: { kind: 'file', size_bytes: 5 * 1024 * 1024, multipart: true } });++ assert.strictEqual(r.status, 413, 'MEDIA_MULTIPART_MAX_MB');++ r = await call('POST', '/api/v2/tiny/objects', { bearer: 'tiny-key', body: { kind: 'file', size_bytes: 2 * 1024 * 1024, multipart: true, part_size: PART } });++ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.quota.exceeded'], 'the tenant quota at init');++ r = await call('POST', O, { body: { kind: 'file', mime_type: 'application/octet-stream', size_bytes: big.length, multipart: true, part_size: PART, filename: 'big.bin', content_hash: sha(big) } });++ assert.strictEqual(r.status, 201, JSON.stringify(r.body));++ const mp = r.body;++ const up = mp.upload.multipart;++ assert.deepStrictEqual([mp.upload.method, mp.upload.url, up.parts_expected, up.part_size, up.total_size, up.status], ['multipart', null, 3, PART, big.length, 'active']);++ assert.ok(/^mup_/.test(up.upload_id) && /^mp1\./.test(up.token));++ const partUrl = (n) => local(up.part_url_template.replace('{part_number}', String(n)));++ const slice = (n) => big.subarray((n - 1) * PART, Math.min(big.length, n * PART));++++ r = await call('PUT', partUrl(1), { bearer: null, raw: slice(1) });++ assert.deepStrictEqual([r.status, r.body.part_number, r.body.sha256], [200, 1, sha(slice(1))]);++ r = await call('PUT', `${O}/${mp.id}/content`, { raw: big.subarray(0, 10) });++ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.object.too_large'], 'no single PUT for a multipart-sized object');++ r = await call('PUT', partUrl(2), { bearer: null, raw: slice(2).subarray(0, 1000) });++ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.upload.part_size_mismatch'], 'every part has an exact size');++ r = await call('PUT', partUrl(2), { bearer: null, raw: slice(2), headers: { 'x-content-sha256': sha(Buffer.from('other')) } });++ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.upload.part_hash_mismatch']);++ r = await call('PUT', partUrl(4), { bearer: null, raw: slice(3) });++ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.upload.invalid_part']);++ r = await call('PUT', partUrl(2).replace(/token=[^&]+/, 'token=mp1.99999999999.forged'), { bearer: null, raw: slice(2) });++ assert.strictEqual(r.status, 401);++++ // A dropped connection mid-part: nothing of it is kept; the session says what is missing.++ await new Promise((resolve) => {++ const u = new URL(partUrl(2));++ const req = http.request({ host: u.hostname, port: u.port, path: u.pathname + u.search, method: 'PUT', headers: { 'content-length': slice(2).length } });++ req.on('error', () => resolve());++ req.write(slice(2).subarray(0, 200000));++ setTimeout(() => { req.destroy(); setTimeout(resolve, 150); }, 100);++ });++ r = await call('GET', local(up.status_url), { bearer: null });++ assert.deepStrictEqual([r.status, r.body.parts.map((p) => p.part_number), r.body.missing, r.body.received_bytes], [200, [1], [2, 3], PART]);++ assert.deepStrictEqual(fs.readdirSync(path.join(process.env.OBJECTS_PATH, '.parts', up.upload_id)), ['1'], 'no half part on disk');++ // Resume: send what is missing (part 3 with the app key instead of the session token).++ r = await call('PUT', partUrl(2), { bearer: null, raw: slice(2), headers: { 'x-content-sha256': sha(slice(2)) } });++ assert.strictEqual(r.status, 200);++ r = await call('PUT', `${O}/${mp.id}/multipart/${up.upload_id}/parts/3`, { raw: slice(3) });++ assert.strictEqual(r.status, 200, 'the tenant credential works as well as the session token');++ r = await call('GET', `${O}/${mp.id}/multipart/${up.upload_id}`);++ assert.deepStrictEqual([r.body.missing, r.body.received_bytes], [[], big.length]);++++ r = await call('POST', local(up.complete_url), { bearer: null, body: { parts: [{ part_number: 1, sha256: sha(slice(2)) }] } });++ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.upload.part_hash_mismatch'], "the client's part list must match");++ assert.strictEqual(multipart.getSession(up.upload_id).status, 'active', 'and the session stays open');++ r = await call('POST', local(up.complete_url), { bearer: null, body: { parts: [1, 2, 3].map((n) => ({ part_number: n, sha256: sha(slice(n)) })) } });++ assert.strictEqual(r.status, 200, JSON.stringify(r.body));++ assert.deepStrictEqual([r.body.lifecycle_status, r.body.size_bytes, r.body.content_hash], ['ready', big.length, sha(big)]);++ assert.ok(!fs.existsSync(path.join(process.env.OBJECTS_PATH, '.parts', up.upload_id)), 'parts removed');++ assert.strictEqual(multipart.getSession(up.upload_id).status, 'completed');++ assert.deepStrictEqual(fs.readFileSync(model.objectFilePath(model.getObject(mp.id))), big, 'the assembled object is the original');++ r = await call('GET', `${O}/${mp.id}/download`);++ r = await call('GET', local(r.body.url), { bearer: null });++ assert.deepStrictEqual([r.status, sha(Buffer.from(await (await fetch(local((await call('GET', `${O}/${mp.id}/download`)).body.url))).arrayBuffer()))], [200, sha(big)]);++ r = await call('POST', local(up.complete_url), { bearer: null });++ assert.strictEqual(r.status, 409, 'completing twice');++ console.log('✅ multipart: exact parts, sha256 per part, resumable after a dropped connection, assembled and verified');++++ // Declare the size later; abort; a session for another object does not open this one; expiry.++ r = await call('POST', O, { body: { kind: 'file' } });++ const later = r.body.id;++ r = await call('POST', `${O}/${later}/multipart`, { body: { part_size: PART } });++ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.object.invalid'], 'a size is needed');++ r = await call('POST', `${O}/${later}/multipart`, { body: { size_bytes: PART + 10, part_size: PART } });++ assert.strictEqual(r.status, 201);++ const s2 = r.body;++ r = await call('PUT', `${O}/${later}/content`, { raw: Buffer.alloc(PART + 10) });++ assert.deepStrictEqual([r.status, r.body.code], [409, 'media.upload.multipart_active']);++ r = await call('PUT', local(s2.part_url_template.replace('{part_number}', '1')).replace(later, a.id).replace(s2.upload_id, up.upload_id), { bearer: null, raw: Buffer.alloc(PART) });++ assert.strictEqual(r.status, 401, "a session token opens only its own session's object");++ r = await call('PUT', local(s2.part_url_template.replace('{part_number}', '1')), { bearer: null, raw: Buffer.alloc(PART) });++ assert.strictEqual(r.status, 200);++ r = await call('DELETE', local(s2.abort_url), { bearer: null });++ assert.deepStrictEqual([r.status, r.body.status], [200, 'aborted']);++ assert.ok(!fs.existsSync(path.join(process.env.OBJECTS_PATH, '.parts', s2.upload_id)));++ assert.strictEqual(model.getObject(later).lifecycle_status, 'uploading', 'the object can be sent again');++ r = await call('PUT', local(s2.part_url_template.replace('{part_number}', '2')), { bearer: null, raw: Buffer.alloc(10) });++ assert.deepStrictEqual([r.status, r.body.code], [409, 'media.upload.not_active']);++ r = await call('POST', `${O}/${later}/multipart`, { body: { part_size: PART } });++ const s3 = r.body;++ await call('PUT', local(s3.part_url_template.replace('{part_number}', '1')), { bearer: null, raw: Buffer.alloc(PART) });++ db.run("UPDATE media_uploads SET expires_at = datetime('now', '-1 minute') WHERE id = ?", [s3.upload_id]);++ fs.mkdirSync(path.join(process.env.OBJECTS_PATH, '.parts', 'mup_orphan'), { recursive: true });++ const rec = await require('../server/objects/reconcile').reconcile({ appId: 'live' });++ assert.deepStrictEqual(rec.issues.incomplete_multipart.items.map((i) => [i.upload_id, i.parts_received]), [[s3.upload_id, 1]], 'reconciliation reports an open session past its expiry');++ assert.deepStrictEqual(multipart.purgeExpired(), { expired: 1, orphan_dirs: 1 });++ assert.strictEqual(multipart.getSession(s3.upload_id).status, 'expired');++ assert.deepStrictEqual(fs.readdirSync(path.join(process.env.OBJECTS_PATH, '.parts')), [], 'expired and orphan parts removed');++ console.log('✅ multipart: size declared later, abort, per-session tokens, single PUT refused while open, expiry purge');++++ server.close();++ console.log('\nobjects multipart + presigned: all checks passed');++ process.exit(0);++})().catch((err) => { console.error(err); process.exit(1); });nnew file mode 100644@@@ -0,0 +1,187 @@++'use strict';++// scripts/r2-eviction-drill.js against a fake S3 (B2 and R2 buckets on one local endpoint, spoken to by++// the real AWS SDK through vod-storage) and a local Media serving /v/:id: the dry run changes nothing;++// the drill refuses when the canonical copy looks wrong, when a hold freezes placement, or when there++// is no R2 copy; --execute evicts the R2 copy, proves /v/:id is served from B2 with the same bytes,++// re-warms R2 and proves it again; a failed re-warm leaves the VOD served from B2; the CLI writes the artifact.++const assert = require('assert');++const fs = require('fs');++const os = require('os');++const path = require('path');++const http = require('http');++const crypto = require('crypto');++const { execFile } = require('child_process');++// The fake S3 lives in this process, so the CLI must run without blocking it (no spawnSync).++const runCli = (args) => new Promise((resolve) => {++ execFile(process.execPath, [path.join(__dirname, '../scripts/r2-eviction-drill.js'), ...args], { env: process.env, encoding: 'utf8' },++ (err, stdout, stderr) => resolve({ status: err ? err.code : 0, stdout, stderr }));++});++++const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-r2drill-'));++const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++++// ── Fake S3: path-style /<bucket>/<key>; HEAD, GET (Range), PUT, DELETE ──++const buckets = { 'b2-bucket': new Map(), 'r2-bucket': new Map() };++const s3log = [];++let refusePutTo = null;++function decodeAwsChunked(buf) {++ const out = [];++ let i = 0;++ for (;;) {++ const nl = buf.indexOf('\r\n', i);++ const size = parseInt(buf.subarray(i, nl).toString().split(';')[0], 16);++ if (!size) break;++ out.push(buf.subarray(nl + 2, nl + 2 + size));++ i = nl + 2 + size + 2;++ }++ return Buffer.concat(out);++}++const s3 = http.createServer((req, res) => {++ const chunks = [];++ req.on('data', (c) => chunks.push(c));++ req.on('end', () => {++ const u = new URL(req.url, 'http://x');++ const [, bucket, ...rest] = decodeURIComponent(u.pathname).split('/');++ const key = rest.join('/');++ const store = buckets[bucket];++ s3log.push([req.method, bucket, key]);++ if (!store) { res.writeHead(404); return res.end(); }++ const obj = store.get(key);++ if (req.method === 'HEAD' || req.method === 'GET') {++ if (!obj) { res.writeHead(404, { 'Content-Type': 'application/xml' }); return res.end(req.method === 'GET' ? '<Error><Code>NoSuchKey</Code></Error>' : undefined); }++ const m = /bytes=(\d+)-(\d+)?/.exec(req.headers.range || '');++ if (m && req.method === 'GET') {++ const start = Number(m[1]), end = Math.min(obj.length - 1, m[2] != null ? Number(m[2]) : obj.length - 1);++ res.writeHead(206, { 'Content-Length': end - start + 1, 'Content-Range': `bytes ${start}-${end}/${obj.length}`, ETag: '"e"' });++ return res.end(obj.subarray(start, end + 1));++ }++ res.writeHead(200, { 'Content-Length': obj.length, ETag: '"e"', 'Content-Type': 'video/webm' });++ return res.end(req.method === 'GET' ? obj : undefined);++ }++ if (req.method === 'PUT') {++ if (refusePutTo === bucket) { res.writeHead(500, { 'Content-Type': 'application/xml' }); return res.end('<Error><Code>InternalError</Code></Error>'); }++ let body = Buffer.concat(chunks);++ if (req.headers['x-amz-decoded-content-length'] || /aws-chunked/.test(req.headers['content-encoding'] || '')) body = decodeAwsChunked(body);++ store.set(key, body);++ res.writeHead(200, { ETag: '"e"' });++ return res.end();++ }++ if (req.method === 'DELETE') { store.delete(key); res.writeHead(204); return res.end(); }++ res.writeHead(405); res.end();++ });++});++++(async () => {++ await new Promise((r) => s3.listen(0, '127.0.0.1', r));++ const endpoint = `http://127.0.0.1:${s3.address().port}`;++ Object.assign(process.env, {++ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),++ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),++ MEDIA_B2_ENDPOINT: endpoint, MEDIA_B2_BUCKET: 'b2-bucket', MEDIA_B2_KEY_ID: 'k', MEDIA_B2_APP_KEY: 's', MEDIA_B2_REGION: 'us-west-004',++ MEDIA_R2_ENDPOINT: endpoint, MEDIA_R2_BUCKET: 'r2-bucket', MEDIA_R2_ACCESS_KEY_ID: 'k', MEDIA_R2_SECRET_ACCESS_KEY: 's',++ AWS_REQUEST_CHECKSUM_CALCULATION: 'WHEN_REQUIRED', AWS_RESPONSE_CHECKSUM_VALIDATION: 'WHEN_REQUIRED',++ });++ const db = require('../server/db/database');++ const model = require('../server/objects/model');++ const storage = require('../server/vod/vod-storage');++ const { runDrill, providerOfUrl } = require('../scripts/r2-eviction-drill');++ const conn = db.getDb();++ require('../server/views/service').ensureSchema();++++ // Media serving /v/:id, as production does.++ const express = require('express');++ const app = express();++ app.use('/', require('../server/public/routes'));++ const media = http.createServer(app);++ await new Promise((r) => media.listen(0, '127.0.0.1', r));++ const baseUrl = `http://127.0.0.1:${media.address().port}`;++++ const bytes = crypto.randomBytes(3 * 1024 * 1024 + 123);++ const addVod = (id, { provider = 'r2', visibility = 'public', size = bytes.length } = {}) => {++ const key = `vods/drill-${id}.webm`;++ conn.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, file_size, is_public, visibility, duration_seconds, storage_provider, storage_key)++ VALUES (?, 'live', 5, 'Drill', ?, ?, ?, ?, 60, ?, ?)`).run(id, `/gone/drill-${id}.webm`, size, visibility === 'private' ? 0 : 1, visibility, provider, key);++ buckets['b2-bucket'].set(key, bytes);++ if (provider === 'r2') buckets['r2-bucket'].set(key, bytes);++ model.sync('vod', id);++ return key;++ };++ const deps = { storage, db, model };++ const opts = (o) => ({ http: true, baseUrl, maxMb: 512, ...o });++ assert.strictEqual(providerOfUrl(`${endpoint}/r2-bucket/vods/x.webm?X-Amz-Signature=1`, storage), 'r2');++ assert.strictEqual(providerOfUrl('https://b2-bucket.s3.example/vods/x.webm', { bucketFor: (p) => `${p}-bucket`, endpointFor: () => 'https://s3.example' }), 'b2');++++ // ── Dry run: checks only ──++ const key1 = addVod(1);++ let art = await runDrill(opts({ vodId: 1 }), deps);++ assert.strictEqual(art.verdict, 'dry-run', art.reason);++ assert.deepStrictEqual(art.steps.map((s) => s.name), ['plan']);++ assert.ok(buckets['r2-bucket'].has(key1) && db.get('SELECT storage_provider FROM vods WHERE id = 1').storage_provider === 'r2', 'nothing moved');++ assert.ok(!s3log.some(([m]) => m === 'PUT' || m === 'DELETE'), 'no writes to either bucket');++ assert.strictEqual(art.checks.first_mib.b2, art.checks.first_mib.r2);++ console.log('✅ dry run: preconditions and copy checks only, nothing changes');++++ // ── Refusals ──++ const key2 = addVod(2);++ buckets['b2-bucket'].set(key2, bytes.subarray(0, 100));++ assert.match((await runDrill(opts({ vodId: 2, execute: true }), deps)).reason, /differ in size/);++ buckets['b2-bucket'].set(key2, Buffer.concat([Buffer.from('X'), bytes.subarray(1)]));++ assert.match((await runDrill(opts({ vodId: 2, execute: true }), deps)).reason, /first MiB/);++ buckets['b2-bucket'].delete(key2);++ assert.match((await runDrill(opts({ vodId: 2, execute: true }), deps)).reason, /B2 canonical copy is missing/);++ assert.ok(buckets['r2-bucket'].has(key2), 'a refused drill deletes nothing');++ addVod(3, { provider: 'b2' });++ assert.match((await runDrill(opts({ vodId: 3, execute: true }), deps)).reason, /not R2/);++ addVod(4);++ const hold = model.placeHold({ object_id: db.get('SELECT object_id FROM vods WHERE id = 4').object_id, kind: 'evidence', reason: 'drill test' });++ assert.match((await runDrill(opts({ vodId: 4, execute: true }), deps)).reason, /retention hold/);++ model.releaseHold(hold.id, 'test');++ addVod(5, { visibility: 'private' });++ assert.match((await runDrill(opts({ vodId: 5, execute: true }), deps)).reason, /private/);++ assert.strictEqual((await runDrill(opts({ pick: true }), deps)).vod_id, 1, '--pick takes the smallest public R2 VOD that is not held');++ console.log('✅ refused: canonical copy wrong size / different bytes / missing, not on R2, held, private over HTTP');++++ // ── Execute: evict, served from B2, re-warm, served from R2 ──++ s3log.length = 0;++ art = await runDrill(opts({ vodId: 1, execute: true }), deps);++ assert.strictEqual(art.verdict, 'pass', `${art.reason} ${JSON.stringify(art.steps, null, 1)}`);++ assert.deepStrictEqual(art.steps.map((s) => [s.name.split(':')[0], s.ok]), [['before', true], ['evict', true], ['from B2', true], ['re-warm', true], ['after', true]]);++ assert.deepStrictEqual([art.steps[0].provider, art.steps[2].provider, art.steps[4].provider], ['r2', 'b2', 'r2'], 'GET /v/1 went R2 -> B2 -> R2');++ assert.deepStrictEqual([art.steps[0].status, art.steps[2].range_status], [302, 206]);++ const sha = crypto.createHash('sha256').update(bytes.subarray(0, 1024 * 1024)).digest('hex');++ assert.ok(art.steps.filter((s) => s.first_mib_sha256).every((s) => s.first_mib_sha256 === sha), 'the same bytes every time');++ assert.ok(s3log.some(([m, b]) => m === 'DELETE' && b === 'r2-bucket') && s3log.some(([m, b]) => m === 'PUT' && b === 'r2-bucket'), 'the R2 copy was really deleted and written back');++ assert.ok(!s3log.some(([m, b]) => (m === 'DELETE' || m === 'PUT') && b === 'b2-bucket'), 'B2 was never written');++ assert.deepStrictEqual(buckets['r2-bucket'].get(key1), bytes, 'R2 holds the same bytes again');++ assert.strictEqual(db.get('SELECT storage_provider FROM vods WHERE id = 1').storage_provider, 'r2');++ assert.deepStrictEqual(art.locations.after_evict.map((l) => l.provider).sort(), ['b2']);++ assert.deepStrictEqual(art.locations.after.map((l) => [l.provider, l.state]).sort(), [['b2', 'present'], ['r2', 'present']]);++ console.log('✅ execute: R2 copy evicted, /v served from B2 with the same bytes, R2 re-warmed and serving again');++++ // ── A failed re-warm leaves the VOD served from B2 ──++ addVod(6);++ refusePutTo = 'r2-bucket';++ art = await runDrill(opts({ vodId: 6, execute: true, http: false }), deps);++ refusePutTo = null;++ assert.strictEqual(art.verdict, 'fail');++ assert.deepStrictEqual(art.steps.map((s) => s.ok), [true, true, true, false]);++ assert.strictEqual(art.steps[0].via, 'resolvePlayback', '--no-http decides in-process');++ assert.strictEqual(db.get('SELECT storage_provider FROM vods WHERE id = 6').storage_provider, 'b2');++ assert.strictEqual((await storage.resolvePlayback(db.get('SELECT * FROM vods WHERE id = 6'))).provider, 'b2', 'still served, from B2');++ console.log('✅ a failed re-warm stops the drill with the VOD served from B2');++++ // ── The CLI writes the artifact ──++ const out = path.join(tmp, 'artifact.json');++ const cli = await runCli(['--vod', '1', '--no-http', '--out', out]);++ assert.strictEqual(cli.status, 0, cli.stderr + cli.stdout);++ const saved = JSON.parse(fs.readFileSync(out, 'utf8'));++ assert.deepStrictEqual([saved.drill, saved.mode, saved.verdict, saved.vod_id], ['r2-eviction', 'dry-run', 'dry-run', 1]);++ const refused = await runCli(['--vod', '3', '--out', path.join(tmp, 'r.json')]);++ assert.strictEqual(refused.status, 2, 'refused -> exit 2');++ console.log('✅ CLI: dry run by default, JSON artifact written, exit codes');++++ media.close();++ s3.close();++ console.log('\nr2 eviction drill: all checks passed');++ process.exit(0);++})().catch((err) => { console.error(err); process.exit(1); });
@@ -68,3 +68,21 @@ MEDIA_R2_SECRET_ACCESS_KEY=
@@ -68,3 +68,21 @@ MEDIA_R2_SECRET_ACCESS_KEY=
# OpenVibe.Events (durable events): vod/clip/object/storage outcomes are also published through
# OpenVibe.Events (durable events): vod/clip/object/storage outcomes are also published through
# Media's outbox with the OAuth client credentials (grant events.event.publish). Unset = off.
# Media's outbox with the OAuth client credentials (grant events.event.publish). Unset = off.
#EVENTS_URL=http://127.0.0.1:4300
#EVENTS_URL=http://127.0.0.1:4300
+
+# ── Object uploads (docs/object-model.md#presigned-uploads, #multipart-uploads) ──
+# MEDIA_SIGNING_SECRET signs presigned upload/download URLs; set it in production.
+#MEDIA_SIGNING_SECRET=
+#MEDIA_OBJECT_MAX_MB=256 # single-part limit; larger objects use multipart
+#MEDIA_UPLOAD_TOKEN_TTL_S=3600 # default presigned upload URL lifetime
+#MEDIA_MULTIPART_MAX_MB=20480
+#MEDIA_MULTIPART_PART_MB=64 # bounds: MEDIA_MULTIPART_MIN_PART_MB=5, MEDIA_MULTIPART_MAX_PART_MB=256
+#MEDIA_MULTIPART_TTL_HOURS=24
+#MEDIA_UPLOAD_MIN_FREE_MB=10240 # free disk kept in reserve by multipart uploads and split/remux jobs
+
+# ── Jobs (docs/object-model.md#jobs) ─────────────────────────
+#MEDIA_JOBS_ENABLED=1
+#MEDIA_JOBS_POLL_MS=5000
+#MEDIA_JOBS_LIGHT_CONCURRENCY=2
+#MEDIA_JOBS_HEAVY_CONCURRENCY=1
+#MEDIA_JOBS_HEAVY_WHILE_RECORDING=0
+#MEDIA_INVARIANT_SCAN_HOURS=24 # the size-invariant validator (proposes split/remux; never runs them)
@@ -41,11 +41,18 @@ server/
@@ -41,11 +41,18 @@ server/
thumbnails/ thumbnail service + /api/v1/:app/thumbnails
thumbnails/ thumbnail service + /api/v1/:app/thumbnails
public/routes.js public /v /c /p /t /f
public/routes.js public /v /c /p /t /f
objects/ canonical object model: model (projections, holds), routes (/api/v2 + /o),
objects/ canonical object model: model (projections, holds), routes (/api/v2 + /o),
- backfill, reconcile, invariant, signing, verify-job (scheduled copy
- verification), copy-report — see docs/object-model.md
+ backfill, reconcile, invariant, signing (presigned URLs), multipart,
+ content-type, verify-job (scheduled copy verification), copy-report —
+ see docs/object-model.md
+ jobs/ media_jobs: queue (states, idempotency, media.job.* events in the state
+ change's transaction), worker (light/heavy lanes, leases, retries,
+ cancellation), routes (/api/v2/:app/jobs), types: thumbnail.regenerate,
+ invariant.scan (proposes split/remux), object.split, object.remux
(openvibe-shared: pinned OpenVibe.Shared v1.0.0 release, installed by npm)
(openvibe-shared: pinned OpenVibe.Shared v1.0.0 release, installed by npm)
scripts/smoke-test.sh end-to-end smoke test (boots a temp instance)
scripts/smoke-test.sh end-to-end smoke test (boots a temp instance)
scripts/backfill-objects.js / reconcile-objects.js / object-invariant.js / no-good-copy-report.js object-model operator tools
scripts/backfill-objects.js / reconcile-objects.js / object-invariant.js / no-good-copy-report.js object-model operator tools
+scripts/media-jobs.js list jobs, run the size-invariant scan (dry run by default), approve/cancel proposals
+scripts/r2-eviction-drill.js evict one VOD's R2 copy, prove B2 serves it, re-warm R2; JSON artifact (dry run by default)
```
```
## Visitor sign-in
## Visitor sign-in
@@ -7,7 +7,10 @@ projection over an object through its `object_id` column. Every existing URL and
@@ -7,7 +7,10 @@ projection over an object through its `object_id` column. Every existing URL and
unchanged. New code talks to objects through the [v2 API](#object-api-v2).
unchanged. New code talks to objects through the [v2 API](#object-api-v2).
Code: `server/objects/` (`model.js`, `routes.js`, `backfill.js`, `reconcile.js`, `invariant.js`,
Code: `server/objects/` (`model.js`, `routes.js`, `backfill.js`, `reconcile.js`, `invariant.js`,
-`signing.js`, `verify-job.js`, `copy-report.js`). Schema: the bottom of `server/db/schema.sql`. Tests: `test/objects-*.test.js`.
+`signing.js`, `verify-job.js`, `copy-report.js`, `multipart.js`, `content-type.js`) and the job system in
+`server/jobs/` (`queue.js`, `worker.js`, `routes.js`, `types.js`, `thumbnail.js`, `invariant-scan.js`,
+`derive.js`). Schema: the bottom of `server/db/schema.sql`. Tests: `test/objects-*.test.js`, `test/jobs*.test.js`,
+`test/r2-eviction-drill.test.js`.
## Tables
## Tables
@@ -17,7 +20,8 @@ Code: `server/objects/` (`model.js`, `routes.js`, `backfill.js`, `reconcile.js`,
@@ -17,7 +20,8 @@ Code: `server/objects/` (`model.js`, `routes.js`, `backfill.js`, `reconcile.js`,
| `media_locations` | one row per provider copy (`UNIQUE(object_id, provider)`): `provider` (`local` `b2` `r2`), `bucket`, `key` (absolute path for local, object key for B2/R2), `storage_class` (`hot` local, `cold` B2, `cache` R2), `state`, `checksum`, `size_bytes`, `verified_at` |
| `media_locations` | one row per provider copy (`UNIQUE(object_id, provider)`): `provider` (`local` `b2` `r2`), `bucket`, `key` (absolute path for local, object key for B2/R2), `storage_class` (`hot` local, `cold` B2, `cache` R2), `state`, `checksum`, `size_bytes`, `verified_at` |
| `media_relationships` | `from_object_id`, `relation`, `to_object_id`, `metadata`. In use: `clip_of` (clip to vod, with start/end), `thumbnail_of` (thumbnail to vod/clip). Reserved: `derived_from`, `screenshot_of` |
| `media_relationships` | `from_object_id`, `relation`, `to_object_id`, `metadata`. In use: `clip_of` (clip to vod, with start/end), `thumbnail_of` (thumbnail to vod/clip). Reserved: `derived_from`, `screenshot_of` |
| `media_variants` | `object_id`, `variant_name`, `derived_object_id`, `recipe`. In use: `thumbnail` |
| `media_variants` | `object_id`, `variant_name`, `derived_object_id`, `recipe`. In use: `thumbnail` |
-| `media_jobs` | generic derivative/maintenance jobs (`job_type`, `status`, `attempts`, `checkpoint`, `error`). **Schema only in this pass:** nothing enqueues or consumes jobs yet |
+| `media_jobs` | derivative/maintenance jobs: `id` (mjob_…), `app_id`, `object_id`, `job_type`, `status`, `idempotency_key`, `params`, `result`, `attempts`/`max_attempts`, `run_after`, `lease_until`, `checkpoint`, `error`/`error_code`, `cancel_requested`, `created_by`, `decided_by`. See [Jobs](#jobs) |
+| `media_uploads`, `media_upload_parts` | multipart upload sessions and the parts received (size, sha256). See [Multipart uploads](#multipart-uploads) |
| `media_holds` | retention holds, see [Holds](#retention-holds) |
| `media_holds` | retention holds, see [Holds](#retention-holds) |
| `media_invariant_violations` | public playback objects over the size policy, see [Invariant](#public-object-size-invariant) |
| `media_invariant_violations` | public playback objects over the size policy, see [Invariant](#public-object-size-invariant) |
@@ -129,9 +133,15 @@ All routes live under `/api/v2/:app/objects`.
@@ -129,9 +133,15 @@ All routes live under `/api/v2/:app/objects`.
| method | path | notes |
| method | path | notes |
|---|---|---|
|---|---|---|
-| POST | `/` | init. `{ kind, visibility (default private), mime_type, size_bytes, filename, content_hash (expected sha256), metadata (≤ 16 KB), user_id }` returns 201 `{ id, object, upload: { method: 'PUT', url, token, expires_at, max_bytes, complete_url } }`. Checks at init: `size_bytes` against `MEDIA_OBJECT_MAX_MB` (413), the quota including reservations (413 `media.quota.exceeded`), and the public-size invariant (422) |
+| POST | `/` | init. `{ kind, visibility (default private), mime_type, size_bytes, filename, content_hash (expected sha256), metadata (≤ 16 KB), user_id, upload_ttl (60-86400 s), multipart, part_size }` returns 201 `{ id, object, upload: { method: 'PUT', url, token, expires_at, max_bytes, content_type, complete_url, multipart_url } }`. `url` is a [presigned PUT URL](#presigned-uploads). With `multipart: true` (required above `MEDIA_OBJECT_MAX_MB`): `method: 'multipart'`, `url: null`, and `upload.multipart` is the new [session](#multipart-uploads). Checks at init: `size_bytes` against `MEDIA_OBJECT_MAX_MB` (single part) or `MEDIA_MULTIPART_MAX_MB` (413), the quota including reservations (413 `media.quota.exceeded`), the public-size invariant (422) and the [content type](#content-types) for the kind (415 `media.object.unsupported_type`) |
+| POST | `/:id/upload-url` | a fresh presigned single-PUT URL for an uploading object (`{ ttl }` 60-86400 s) |
| PUT | `/:id/content` | the bytes, single part. Auth is the upload token (`?token=` from init, or `X-Upload-Token`) or the usual credential. Streams to disk computing sha256. Refuses more than the declared size or the limit (413), a size mismatch (400) and quota overrun (413). Stored at `OBJECTS_PATH/<app>/<id>` with a verified local location. Mounted ahead of the JSON body parser, so any Content-Type is taken as raw bytes; the Content-Type becomes `mime_type` when init set none. Can be repeated while the object is `uploading` |
| PUT | `/:id/content` | the bytes, single part. Auth is the upload token (`?token=` from init, or `X-Upload-Token`) or the usual credential. Streams to disk computing sha256. Refuses more than the declared size or the limit (413), a size mismatch (400) and quota overrun (413). Stored at `OBJECTS_PATH/<app>/<id>` with a verified local location. Mounted ahead of the JSON body parser, so any Content-Type is taken as raw bytes; the Content-Type becomes `mime_type` when init set none. Can be repeated while the object is `uploading` |
-| POST | `/:id/complete` | checks the expected hash (422 `media.object.hash_mismatch`), the invariant and the quota. Sets `ready` and sends the `media.object.uploaded` webhook. Also accepts the upload token |
+| POST | `/:id/complete` | checks the expected hash (422 `media.object.hash_mismatch`), the invariant, the quota and the bytes against the type (415 `media.object.content_mismatch`). Sets `ready` and sends the `media.object.uploaded` webhook. Also accepts the upload token |
+| POST | `/:id/multipart` | start a [multipart session](#multipart-uploads) `{ part_size, size_bytes (when init declared none) }` → 201 |
+| GET | `/:id/multipart/:uploadId` | the session: `parts` received (number, size, sha256), `missing`, `received_bytes`, `expires_at` |
+| PUT | `/:id/multipart/:uploadId/parts/:n` | one part, raw bytes of exactly its size; optional `X-Content-SHA256` |
+| POST | `/:id/multipart/:uploadId/complete` | `{ content_hash, parts: [{ part_number, sha256 }] }` (both optional): assembles, then the checks of `/complete` |
+| DELETE | `/:id/multipart/:uploadId` | abort: parts deleted, the object stays `uploading` |
| GET | `/:id` | metadata: providers and states of each copy, never paths or keys |
| GET | `/:id` | metadata: providers and states of each copy, never paths or keys |
| GET | `/` | cursor list, newest first. `?limit (≤200)&cursor&kind&visibility&status&owner (usr_…)&user_id&include_deleted` returns `{ objects, next_cursor }` |
| GET | `/` | cursor list, newest first. `?limit (≤200)&cursor&kind&visibility&status&owner (usr_…)&user_id&include_deleted` returns `{ objects, next_cursor }` |
| DELETE | `/:id` | soft delete: `lifecycle_status = deleted`, and the bytes are kept for `MEDIA_DELETE_RETENTION_DAYS` (default 30). 409 `media.object.held` under a hold. 409 `media.object.legacy_managed` for projected objects, which are deleted through their v1 route |
| DELETE | `/:id` | soft delete: `lifecycle_status = deleted`, and the bytes are kept for `MEDIA_DELETE_RETENTION_DAYS` (default 30). 409 `media.object.held` under a hold. 409 `media.object.legacy_managed` for projected objects, which are deleted through their v1 route |
@@ -149,10 +159,71 @@ All routes live under `/api/v2/:app/objects`.
@@ -149,10 +159,71 @@ All routes live under `/api/v2/:app/objects`.
- A local copy streams with Range support. Otherwise the route redirects 302 to a presigned R2 URL, then a presigned B2 URL.
- A local copy streams with Range support. Otherwise the route redirects 302 to a presigned R2 URL, then a presigned B2 URL.
- Headers: `X-Content-Type-Options: nosniff` and `X-Robots-Tag: noindex`. Content is `inline` only for images (not SVG), video, audio, PDF and plain text, and `attachment` for everything else.
- Headers: `X-Content-Type-Options: nosniff` and `X-Robots-Tag: noindex`. Content is `inline` only for images (not SVG), video, audio, PDF and plain text, and `attachment` for everything else.
-**Signing.** HMAC-SHA256 over the purpose (`get` or `put`), the object id and the expiry, keyed with
-`MEDIA_SIGNING_SECRET`. A download signature cannot be used as an upload token, or the other way
-round. Without the secret, a random per-process secret is used and a warning is logged, so links
-stop working at the next restart. **Set `MEDIA_SIGNING_SECRET` in production.**
+**Signing.** HMAC-SHA256 keyed with `MEDIA_SIGNING_SECRET` over a purpose and what the link may do:
+downloads (`get`) over the object id and expiry; upload tokens (`put2`) over the tenant, the object,
+its size and the expiry; multipart sessions (`mpart`) over the tenant, the object, the session, its
+total size and the expiry. A signature for one purpose is never valid for another. Without the
+secret, a random per-process secret is used and a warning is logged, so links stop working at the
+next restart. **Set `MEDIA_SIGNING_SECRET` in production.**
+
+### Presigned uploads
+
+The `url` from init (or `POST /:id/upload-url`) is a presigned single-PUT URL:
+`…/objects/<id>/content?token=v2.<exp>.<size>.<mac>`. No other credential is needed, so a browser
+can send the bytes straight to Media. The token is scoped:
+
+- to the **tenant** and the **object**: a token for another object or another tenant never verifies,
+ and the URL must address the object's tenant;
+- to the **size** the object declared (the PUT must be exactly that many bytes). An object created
+ without a size gets a size-0 token: any size up to `MEDIA_OBJECT_MAX_MB`;
+- in **time**: `upload_ttl` / `ttl` (60 s to 24 h, default `MEDIA_UPLOAD_TOKEN_TTL_S`).
+
+The size is part of the signature, so it cannot be edited. Tokens of the earlier form `<exp>.<mac>`
+(object only) are accepted until they expire. These are Media-signed URLs: the bytes land on Media's
+disk, not directly in a bucket.
+
+### Multipart uploads
+
+For objects above `MEDIA_OBJECT_MAX_MB` (up to `MEDIA_MULTIPART_MAX_MB`, default 20 GB), or whenever a
+client asks with `multipart: true`. Code: `server/objects/multipart.js`.
+
+1. **Initiate** at init (`multipart: true`) or with `POST /:id/multipart`. The object must declare its
+ size. `part_size` is `MEDIA_MULTIPART_MIN_PART_MB` (5) to `MEDIA_MULTIPART_MAX_PART_MB` (256) MB,
+ default `MEDIA_MULTIPART_PART_MB` (64), and grows if there would be more than 10,000 parts. The
+ answer carries `upload_id`, `parts_expected`, and a session `token` (`mp1.…`) already in
+ `part_url_template`, `status_url`, `complete_url` and `abort_url`. The quota is checked (the
+ declared size is reserved from init), and there must be free disk for twice the size plus
+ `MEDIA_UPLOAD_MIN_FREE_MB` (507 `media.storage.insufficient`). A new session replaces an open one.
+2. **Upload parts** (`PUT …/parts/<n>`, any order, in parallel). Every part but the last is exactly
+ `part_size` bytes and the last is the rest; any other size is refused (400/413). A part is written
+ to a temp file and renamed when complete, so a dropped connection leaves nothing half-written.
+ Sending a part again replaces it. `X-Content-SHA256` is checked when sent.
+3. **Resume** after a drop: `GET` the session and send the parts in `missing`.
+4. **Complete**: every part must be there (409 `media.upload.parts_missing` lists `missing`); a
+ `parts` list with sha256 values must match what was received. The parts are concatenated into
+ `OBJECTS_PATH/<app>/<id>` while hashing, the parts are deleted, and the checks of `/complete`
+ follow (expected hash, invariant, quota, content). If one of those fails, the object stays
+ `uploading` with its content and can be sent again.
+5. **Abort** (`DELETE`) deletes the parts; the object stays `uploading`. A single-part PUT is refused
+ while a session is open (409 `media.upload.multipart_active`).
+
+Sessions expire after `MEDIA_MULTIPART_TTL_HOURS` (24): the hourly purge deletes their parts and any
+part directory no session owns. `media_uploads_open` on `/metrics` counts open sessions, and
+reconciliation reports sessions left open past their expiry (`incomplete_multipart`).
+
+### Content types
+
+Code: `server/objects/content-type.js`.
+
+- **The type must suit the kind** (415 `media.object.unsupported_type`, at init or at the PUT when
+ init declared none): `vod` video/* or audio/*, `clip` video/*, `thumbnail` / `screenshot` / `avatar`
+ image/* but not SVG. `file` and `asset` take any well-formed type.
+- **Inline-served bytes must be what their type says** (415 `media.object.content_mismatch` at
+ complete): `/o/:id` serves image/*, video/*, audio/* and PDF inline, so for those types (and every
+ kind with a rule) the first bytes must match a known signature of that family: JPEG, PNG, GIF, WebP,
+ BMP, AVIF/HEIC; Matroska/WebM, MP4/MOV, Ogg, AVI, MPEG-TS; MP3, WAV, FLAC, M4A; PDF.
+- Everything else is served as an attachment with `nosniff`, so its declared type is taken as given.
+ The declared type wins over the PUT's `Content-Type` (declared at init: the PUT header is ignored).
**Quota.** `apps.quota_bytes` is compared against v1 file bytes plus native objects that are
**Quota.** `apps.quota_bytes` is compared against v1 file bytes plus native objects that are
`uploading` (their declared size counts as a reservation) or `ready`. Soft-deleted objects do not
`uploading` (their declared size counts as a reservation) or `ready`. Soft-deleted objects do not
@@ -204,9 +275,10 @@ The exit code is 1 when issues were found.
@@ -204,9 +275,10 @@ The exit code is 1 when issues were found.
| `orphan_location` | a location row whose object does not exist |
| `orphan_location` | a location row whose object does not exist |
| `deleted_publicly_reachable` | a deleted object whose inherited row still serves it, or a thumbnail of a deleted object that is still served by name |
| `deleted_publicly_reachable` | a deleted object whose inherited row still serves it, or a thumbnail of a deleted object that is still served by name |
| `missing_projection` | an inherited row with no object yet (run the backfill) |
| `missing_projection` | an inherited row with no object yet (run the backfill) |
+| `incomplete_multipart` | a multipart session still open past its expiry (the hourly purge should have removed it) |
**Not covered yet:** orphan *provider* objects (bucket keys with no row, which needs bucket
**Not covered yet:** orphan *provider* objects (bucket keys with no row, which needs bucket
-listing) and incomplete multipart uploads. `server/objects/reconcile.js` takes a `head` function,
+listing). `server/objects/reconcile.js` takes a `head` function,
so tests run it against a fake provider.
so tests run it against a fake provider.
## Scheduled verification
## Scheduled verification
@@ -260,9 +332,124 @@ A *public playback object* is a `ready` vod or clip whose visibility is `public`
@@ -260,9 +332,124 @@ A *public playback object* is a `ready` vod or clip whose visibility is `public`
Lists every public playback object above target, largest first, records the rows, and resolves rows that no longer apply. `--dry-run` writes nothing.
Lists every public playback object above target, largest first, records the rows, and resolves rows that no longer apply. `--dry-run` writes nothing.
-**Nothing is re-encoded in this pass.** Most whole-stream VODs are far above 500 MB. They will show
-up as violations until the segment-native model lands (roadmap W4 deliverable 7, capture in W7),
-which splits playback into objects that fit.
+- **Validator job.** `invariant.scan` (see [Jobs](#jobs)) runs per tenant every
+ `MEDIA_INVARIANT_SCAN_HOURS` (24) and on demand. It records the violations and **proposes** one job
+ per public playback object above the max: `object.split` into stream-copy parts of about the target
+ size when the duration is known, `object.remux` when it is not (the remux writes the duration, so a
+ split can be planned next). **Proposals never run by themselves**: they wait in status `proposed`
+ until the owner approves or cancels them. A cancelled proposal is not proposed again; a proposal
+ whose object stopped violating (made private, deleted) is withdrawn. `scripts/media-jobs.js scan`
+ shows what would be proposed without writing anything.
+
+**Nothing is re-encoded, and nothing is made private automatically.** A split makes private parts next
+to the unchanged source; the source stays a violation until its owner decides (make the parts public
+and the source private, or leave it). The segment-native model (roadmap W4 deliverable 7, capture in
+W7) is what will make every new recording fit.
+
+## Jobs
+
+Code: `server/jobs/`. Tests: `test/jobs.test.js`, `test/jobs-invariant.test.js`.
+
+**States.** `proposed` (waiting for the owner; the worker never takes it) → `queued` → `running` →
+`succeeded` | `failed` | `cancelled`. A failed attempt goes back to `queued` with `run_after` (backoff
+30 s, 2 min, 8 min, …, at most 1 h) until `max_attempts`; a handler can mark a failure permanent.
+
+**Events.** Every state change and its `media.job.<transition>` event (`proposed`, `queued`,
+`started`, `retrying`, `succeeded`, `failed`, `cancelled`; subject `job <id>`, payload the job)
+commit in one SQLite transaction through Media's outbox, the same rule as the outcome events
+(`webhooks.announce()`): no change without its event, no event for a change that rolled back.
+Progress events are `low` priority; outcomes and proposals are `important`. Job events go to
+OpenVibe.Events only, not to app webhooks.
+
+**Idempotency.** `Idempotency-Key` (or `idempotency_key`) is unique per tenant: the same key with the
+same request (type, object, params) answers with the job it made (`Idempotent-Replayed: true`); a
+different request under a used key is 409 `media.job.idempotency_conflict`. At most 50 jobs per tenant
+may be queued or running (429 `media.job.too_many`).
+
+**Cancellation, by the owner.** A proposed or queued job is cancelled at once. A running job gets
+`cancel_requested`; the worker aborts it (its ffmpeg is killed) and it ends `cancelled`. Finished
+jobs answer 409. The owner is the tenant (app key, service token for the namespace, or the project's
+app token); a caller acting for one of the app's users (`X-OV-User-Id`) sees and decides only the jobs
+it created and the jobs on objects it owns.
+
+**Worker.** In-process, polling every `MEDIA_JOBS_POLL_MS`, in two lanes: `light`
+(`MEDIA_JOBS_LIGHT_CONCURRENCY`, 2) and `heavy` (`MEDIA_JOBS_HEAVY_CONCURRENCY`, 1; waits while a
+recording runs unless `MEDIA_JOBS_HEAVY_WHILE_RECORDING=1`). A running job holds a lease
+(`MEDIA_JOBS_LEASE_S`) renewed by a heartbeat; at start, jobs the previous process left `running` are
+requeued (or failed when out of attempts). Handlers checkpoint progress and resume from it.
+`MEDIA_JOBS_ENABLED=0` stops the worker. Finished thumbnail jobs are pruned after
+`MEDIA_JOBS_RETENTION_DAYS`; other jobs are kept.
+
+| type | lane | what it does |
+|---|---|---|
+| `thumbnail.regenerate` | light | (Re)generates a vod/clip thumbnail from its media (local file or presigned cloud copy). `params { kind, id }`, or an object that is a projected vod/clip. Result `{ url }` |
+| `invariant.scan` | light | The [size-invariant validator](#public-object-size-invariant): records violations, proposes `object.split` / `object.remux`, withdraws moot proposals. Tenant-wide (no object) |
+| `object.split` | heavy | Stream-copies a vod/clip (or a video/audio object) into parts (`params.parts` 2-1000 or `segment_seconds`), each a new **private** object with a `derived_from` relationship (`job_id`, `part`, `start_seconds`, `duration_seconds`). Cuts land on keyframes, so neighbouring parts can overlap slightly. The source is never changed. Checkpointed per part |
+| `object.remux` | heavy | Stream-copy remux of the whole source (seek index, duration, MP4 faststart) into one new private object; also the source's `remux` variant |
+
+Split and remux refuse a source that is not ready, a tenant quota the output would exceed, and too
+little free disk (the source size plus `MEDIA_UPLOAD_MIN_FREE_MB`; retried after 30 min).
+
+**The v1 thumbnail route runs as a job.** `POST /api/v1/:app/thumbnails/:kind/:id` without an image
+queues `thumbnail.regenerate` (joining one already queued or running for the same item, so concurrent
+requests share one ffmpeg run), runs it at once and answers `{ url, job_id }` as before; 404 when
+there is no media, 500 when no frame came out. `?async=1` answers 202 `{ job }` without waiting.
+
+**API** (`/api/v2/:app/jobs`, same auth as the objects API; writes need `media.object.upload`, reads
+`media.object.read`):
+
+| method | path | notes |
+|---|---|---|
+| GET | `/` | cursor list, newest first: `?status&type&object_id&limit (≤200)&cursor` → `{ jobs, next_cursor }` |
+| POST | `/` | `{ type, object_id, params, max_attempts }` + `Idempotency-Key` → 202 `{ job }` (200 on a replay) |
+| GET | `/:jobId` | `{ job }` |
+| POST | `/:jobId/approve` | a proposal → `queued` (409 `media.job.not_proposed` otherwise) |
+| POST | `/:jobId/cancel`, DELETE `/:jobId` | 200 cancelled, 202 while a running job stops, 409 when finished |
+
+**Operator script.**
+
+```
+node scripts/media-jobs.js list [--status proposed] [--type object.split] [--app live] [--json]
+node scripts/media-jobs.js show <job id>
+node scripts/media-jobs.js scan [--app live] [--apply] [--json] # dry run unless --apply
+node scripts/media-jobs.js approve <job id>... [--by <who>]
+node scripts/media-jobs.js cancel <job id>... [--by <who>] [--reason <text>]
+```
+
+It changes the database directly (the service's worker picks approved jobs up on its next poll) and,
+when the service's outbox is on, queues each change's event in the same transaction. It never runs a
+job. `media_jobs{type,status}` on `/metrics` counts jobs.
+
+## R2 eviction drill
+
+The exit criterion "the canonical B2 object survives R2 eviction", as a repeatable drill with a kept
+artifact. Code: `scripts/r2-eviction-drill.js`; test: `test/r2-eviction-drill.test.js` (a fake S3 and
+a local Media, through the real storage engine).
+
+```
+node scripts/r2-eviction-drill.js (--vod <id> | --pick) [--execute] [--base-url https://openvibe.media]
+ [--no-http] [--max-mb 512] [--out artifact.json] [--json]
+```
+
+- **Dry run by default:** it checks the preconditions and both copies, and writes the artifact. Nothing moves.
+- **Preconditions** (else it refuses, exit 2): the VOD is served from R2 (`storage_provider = r2`), is
+ not recording, is not held (a hold freezes placement), and is public or unlisted for the HTTP
+ checks; B2 and R2 are configured; the B2 and R2 copies have the same size (and the recorded size)
+ and the same first MiB. Evicting the cache in front of a corrupt canonical copy would lose the only
+ good one. `--pick` takes the smallest eligible VOD up to `--max-mb`.
+- **`--execute`**, through the storage engine's own tier moves: (1) `GET <base>/v/<id>?raw=1`
+ redirects to R2 and serves the expected first MiB; (2) `demoteFromR2` deletes the R2 copy (after
+ checking B2) and flips the row to `b2`; (3) R2 answers 404, and `/v/<id>` redirects to B2 serving the
+ same first MiB; (4) `promoteToR2` copies B2 back to R2 (verified by HEAD) and flips the row to `r2`;
+ (5) `/v/<id>` redirects to R2 again with the same bytes, and `media_locations` shows B2 and R2 present.
+- A failed step stops the drill (exit 1). After the eviction the VOD stays served from B2, which the
+ drill has just proven works. `--no-http` checks the playback decision in-process instead of
+ `GET /v`, which otherwise counts as one view from the drill's host.
+- **Artifact:** JSON (default `data/drills/r2-eviction-<vod>-<time>.json`) with every step, the HEAD
+ sizes, the first-MiB sha256 of each copy and of what was served, `media_locations` before, after the
+ eviction and after, and the verdict (`pass`, `fail`, `refused`, `dry-run`).
+
+Not run against production yet: it needs the owner's go-ahead.
## Configuration
## Configuration
@@ -273,7 +460,18 @@ which splits playback into objects that fit.
@@ -273,7 +460,18 @@ which splits playback into objects that fit.
| `MEDIA_DELETE_RETENTION_DAYS` | 30 | soft-deleted native bytes are kept this long |
| `MEDIA_DELETE_RETENTION_DAYS` | 30 | soft-deleted native bytes are kept this long |
| `MEDIA_SIGNING_SECRET` | *(random per process)* | HMAC key for signed URLs and upload tokens |
| `MEDIA_SIGNING_SECRET` | *(random per process)* | HMAC key for signed URLs and upload tokens |
| `MEDIA_SIGNED_URL_TTL_S` | 300 | default signed download lifetime |
| `MEDIA_SIGNED_URL_TTL_S` | 300 | default signed download lifetime |
-| `MEDIA_UPLOAD_TOKEN_TTL_S` | 3600 | upload token lifetime |
+| `MEDIA_UPLOAD_TOKEN_TTL_S` | 3600 | default presigned upload URL lifetime (60 s to 24 h per request) |
+| `MEDIA_MULTIPART_MAX_MB` | 20480 | largest object a multipart upload may declare |
+| `MEDIA_MULTIPART_MIN_PART_MB` / `_MAX_PART_MB` / `_PART_MB` | 5 / 256 / 64 | part size bounds and default |
+| `MEDIA_MULTIPART_TTL_HOURS` | 24 | an unfinished multipart session is purged after this |
+| `MEDIA_UPLOAD_MIN_FREE_MB` | 10240 | free disk kept in reserve by multipart uploads and split/remux jobs |
+| `MEDIA_JOBS_ENABLED` | on | `0` stops the job worker |
+| `MEDIA_JOBS_POLL_MS` | 5000 | worker poll interval |
+| `MEDIA_JOBS_LIGHT_CONCURRENCY` / `_HEAVY_CONCURRENCY` | 2 / 1 | jobs per lane |
+| `MEDIA_JOBS_HEAVY_WHILE_RECORDING` | off | `1` lets split/remux run while a recording is being written |
+| `MEDIA_JOBS_LEASE_S` | 120 | running-job lease (renewed by the heartbeat) |
+| `MEDIA_INVARIANT_SCAN_HOURS` | 24 | the size-invariant validator's schedule per tenant (`0` = on demand only) |
+| `MEDIA_JOBS_RETENTION_DAYS` | 30 | finished thumbnail jobs are pruned after this |
| `MEDIA_PUBLIC_OBJECT_{MAX,TARGET,WARN}_MB` | 500 / 256 / 384 | invariant thresholds |
| `MEDIA_PUBLIC_OBJECT_{MAX,TARGET,WARN}_MB` | 500 / 256 / 384 | invariant thresholds |
| `MEDIA_VERIFY_ENABLED` | on | `0` turns the scheduled verification off |
| `MEDIA_VERIFY_ENABLED` | on | `0` turns the scheduled verification off |
| `MEDIA_VERIFY_INTERVAL_MIN` | 10 | minutes between verification runs |
| `MEDIA_VERIFY_INTERVAL_MIN` | 10 | minutes between verification runs |
@@ -284,10 +482,11 @@ which splits playback into objects that fit.
@@ -284,10 +482,11 @@ which splits playback into objects that fit.
## Not in this pass
## Not in this pass
-- Multipart uploads and presigned direct-to-bucket uploads. Native bytes are stored locally.
+- Presigned direct-to-bucket uploads. Presigned URLs are Media-signed and native bytes are stored locally.
- Tiering of native objects to B2/R2.
- Tiering of native objects to B2/R2.
- Copy/move aliases, lifecycle rules and an S3-compatible façade.
- Copy/move aliases, lifecycle rules and an S3-compatible façade.
-- A `media_jobs` worker.
-- Orphan-bucket-object and incomplete-multipart detection.
+- Clip cutting and the finalize remux still run inline, not as jobs (thumbnail regeneration was the first
+ operation moved onto the job system).
+- Orphan-bucket-object detection.
- The segment-native timeline.
- The segment-native timeline.
- App assets (emotes and sounds, the `assets` table) are not projected yet. The `asset` kind exists for v2 uploads.
- App assets (emotes and sounds, the `assets` table) are not projected yet. The `asset` kind exists for v2 uploads.
new file mode 100644
new file mode 100644
@@ -0,0 +1,85 @@
@@ -0,0 +1,85 @@
+#!/usr/bin/env node
+'use strict';
+/**
+ * Media jobs, for the operator (docs/object-model.md#jobs). Works on the service's database directly;
+ * the running service's worker picks up approved jobs on its next poll. When the service's Events
+ * outbox is on (the event_outbox table exists), every state change made here queues its
+ * media.job.* event there too, in the same transaction, and the service relays it.
+ *
+ * node scripts/media-jobs.js list [--status proposed] [--type object.split] [--app live] [--limit 50] [--json]
+ * node scripts/media-jobs.js show <job id> [--json]
+ * node scripts/media-jobs.js scan [--app live] [--apply] [--json]
+ * the size-invariant validator; a dry run unless --apply (which records violations and proposals)
+ * node scripts/media-jobs.js approve <job id>... [--by <who>] proposed -> queued
+ * node scripts/media-jobs.js cancel <job id>... [--by <who>] [--reason <text>]
+ * (every command takes --db ./data/media.db)
+ *
+ * Nothing here runs a job. Exit 0 on success, 1 when a named job could not be decided, 2 on usage errors.
+ */
+const path = require('path');
+
+const args = process.argv.slice(2);
+const has = (n) => args.includes(`--${n}`);
+const arg = (n, d) => { const i = args.indexOf(`--${n}`); return i >= 0 ? args[i + 1] : d; };
+const VALUE_FLAGS = ['status', 'type', 'app', 'limit', 'by', 'reason', 'db'];
+const positional = args.filter((a, i) => !a.startsWith('--') && !(i > 0 && VALUE_FLAGS.includes(String(args[i - 1]).replace(/^--/, ''))));
+if (arg('db')) process.env.DB_PATH = path.resolve(arg('db'));
+
+const db = require('../server/db/database');
+const events = require('../server/events');
+const queue = require('../server/jobs/queue');
+const scan = require('../server/jobs/invariant-scan');
+
+function usage(msg) {
+ if (msg) console.error(msg);
+ console.error('usage: media-jobs.js list|show|scan|approve|cancel … (see the header of this file)');
+ process.exit(2);
+}
+
+function line(j) {
+ const size = j.params && j.params.source_size_bytes ? ` ${(j.params.source_size_bytes / 1048576).toFixed(0)} MB` : '';
+ return `${j.id} ${j.status.padEnd(9)} ${j.type.padEnd(20)} ${j.app_id.padEnd(8)} ${j.object_id || '-'}${size}${j.error ? ` (${j.error})` : ''}`;
+}
+
+(async () => {
+ const [cmd, ...ids] = positional;
+ const by = arg('by', `operator:${require('os').userInfo().username}`);
+ events.initWriter();
+ let exit = 0;
+ if (cmd === 'list') {
+ const apps = arg('app') ? [arg('app')] : db.all('SELECT DISTINCT app_id FROM media_jobs ORDER BY app_id').map(r => r.app_id);
+ const out = [];
+ for (const a of apps) out.push(...queue.list(a, { status: arg('status'), type: arg('type'), limit: arg('limit', 50) }).jobs.map(queue.jobPublic));
+ console.log(has('json') ? JSON.stringify({ jobs: out, counts: queue.counts() }, null, 2) : [...out.map(line), `counts: ${JSON.stringify(queue.counts())}`].join('\n'));
+ } else if (cmd === 'show') {
+ const j = queue.jobPublic(queue.get(ids[0]));
+ if (!j) usage(`no job ${ids[0]}`);
+ console.log(JSON.stringify(j, null, 2));
+ } else if (cmd === 'scan') {
+ const apps = arg('app') ? [arg('app')] : db.all("SELECT DISTINCT app_id FROM media_objects WHERE kind IN ('vod', 'clip') ORDER BY app_id").map(r => r.app_id);
+ const reports = {};
+ for (const a of apps) reports[a] = scan.scanTenant(a, { dryRun: !has('apply') });
+ if (has('json')) console.log(JSON.stringify(reports, null, 2));
+ else {
+ for (const [a, r] of Object.entries(reports)) {
+ console.log(`${a}: ${r.violations} public object(s) over ${(r.thresholds.maxBytes / 1048576).toFixed(0)} MB; ${r.dry_run ? 'would propose' : 'proposed'} ${r.proposed}, already proposed ${r.already_proposed}, ${r.dry_run ? 'would withdraw' : 'withdrawn'} ${r.withdrawn} ${JSON.stringify(r.by_type)}`);
+ }
+ if (!has('apply')) console.log('(dry run: nothing recorded; --apply records the violations and the proposals)');
+ }
+ } else if (cmd === 'approve' || cmd === 'cancel') {
+ if (!ids.length) usage(`${cmd}: name at least one job`);
+ for (const id of ids) {
+ const j = queue.get(id);
+ if (!j) { console.error(`${id}: no such job`); exit = 1; continue; }
+ if (cmd === 'approve') {
+ const out = queue.approve(id, { by });
+ if (out) console.log(`${id}: queued`); else { console.error(`${id}: is ${j.status}, not a proposal`); exit = 1; }
+ } else {
+ const out = queue.cancel(id, { by, reason: arg('reason', null) });
+ if (out.finished) { console.error(`${id}: already ${out.job.status}`); exit = 1; } else console.log(`${id}: ${out.pending ? 'cancel requested (running)' : 'cancelled'}`);
+ }
+ }
+ } else usage(cmd ? `unknown command ${cmd}` : null);
+ db.close();
+ process.exit(exit);
+})().catch((err) => { console.error(err); process.exit(1); });
new file mode 100644
new file mode 100644
@@ -0,0 +1,231 @@
@@ -0,0 +1,231 @@
+#!/usr/bin/env node
+'use strict';
+/**
+ * R2 eviction drill: the canonical B2 copy survives losing the R2 hot-cache copy (roadmap W4 exit
+ * criterion; docs/object-model.md#r2-eviction-drill). Dry run unless --execute.
+ *
+ * node scripts/r2-eviction-drill.js (--vod <id> | --pick) [--execute] [--base-url https://openvibe.media]
+ * [--no-http] [--max-mb 512] [--out artifact.json] [--json] [--db ./data/media.db]
+ *
+ * The VOD must be served from R2 today (storage_provider r2: a B2 canonical copy plus an R2 cache copy),
+ * not recording, not under a retention hold (a hold freezes placement), and public or unlisted unless
+ * --no-http. --pick chooses the smallest such VOD up to --max-mb. Before anything changes, the B2 and R2
+ * copies must have the same size (and the vods row's, when it has one) and the same first MiB: evicting
+ * the cache in front of a corrupt canonical copy would lose the only good one, so the drill refuses.
+ *
+ * With --execute, through the storage engine's own tier moves (server/vod/vod-storage.js):
+ * 1. before HEAD B2 + R2; GET <base>/v/<id>?raw=1 redirects to R2 and serves the same first MiB
+ * 2. evict demoteFromR2(id): B2 canonical checked, the R2 copy deleted, the row flipped to b2
+ * 3. from B2 HEAD R2 is 404; GET /v/<id> now redirects to B2, which serves the same first MiB
+ * 4. re-warm promoteToR2(id): B2 copied back to R2 (verified by HEAD), the row flipped to r2
+ * 5. after GET /v/<id> redirects to R2 again with the same first MiB; media_locations says b2 and r2 present
+ * A failed step stops the drill; the VOD then stays where the last successful step left it (after the
+ * eviction that is B2, which still serves it). --no-http checks the playback decision in-process
+ * (resolvePlayback) instead of GET /v, which otherwise counts as one view from this host.
+ *
+ * Writes a JSON artifact (--out, default ./data/drills/r2-eviction-<vod>-<time>.json) with every step,
+ * the copies' sizes and first-MiB sha256, media_locations before/after, and the verdict.
+ * Exit 0 = pass (or a dry run whose checks passed), 1 = a step failed, 2 = refused (preconditions).
+ * Never run it against production without the owner's go-ahead.
+ */
+const fs = require('fs');
+const path = require('path');
+const crypto = require('crypto');
+
+const MIB = 1024 * 1024;
+
+function parseArgs(argv) {
+ const args = argv.slice(2);
+ const has = (n) => args.includes(`--${n}`);
+ const arg = (n, d) => { const i = args.indexOf(`--${n}`); return i >= 0 ? args[i + 1] : d; };
+ return {
+ vodId: arg('vod') != null ? Number(arg('vod')) : null,
+ pick: has('pick'),
+ execute: has('execute'),
+ http: !has('no-http'),
+ baseUrl: arg('base-url', null),
+ maxMb: Number(arg('max-mb', 512)),
+ out: arg('out', null),
+ json: has('json'),
+ db: arg('db', null),
+ };
+}
+
+/** Which provider a presigned/redirect URL points at (path-style endpoint or bucket.host). */
+function providerOfUrl(url, storage) {
+ let u;
+ try { u = new URL(url); } catch { return null; }
+ for (const p of ['r2', 'b2']) {
+ const bucket = storage.bucketFor(p);
+ const ep = storage.endpointFor ? storage.endpointFor(p) : null;
+ if (!bucket) continue;
+ const host = ep ? new URL(ep).host : null;
+ if (host && u.host === host && u.pathname.startsWith(`/${bucket}/`)) return p;
+ if (host && u.host === `${bucket}.${host}`) return p;
+ if (!host && (u.host.startsWith(`${bucket}.`) || u.pathname.startsWith(`/${bucket}/`))) return p;
+ }
+ return null;
+}
+
+async function firstMibSha(url, fetchImpl) {
+ const res = await fetchImpl(url, { headers: { Range: `bytes=0-${MIB - 1}` } });
+ if (res.status !== 206 && res.status !== 200) throw new Error(`GET ${new URL(url).host}: HTTP ${res.status}`);
+ const body = Buffer.from(await res.arrayBuffer());
+ const bytes = res.status === 200 ? body.subarray(0, MIB) : body;
+ return { status: res.status, sha256: crypto.createHash('sha256').update(bytes).digest('hex'), bytes: bytes.length, content_range: res.headers.get('content-range') };
+}
+
+/**
+ * runDrill(opts, deps) -> artifact. deps: { storage (vod-storage), db, model, fetch, now } — injectable
+ * so test/r2-eviction-drill.test.js runs it against a fake S3 and a local Media.
+ */
+async function runDrill(opts, deps) {
+ const { storage, db, model } = deps;
+ const fetchImpl = deps.fetch || globalThis.fetch;
+ const nowIso = () => new Date((deps.now || Date.now)()).toISOString();
+ const art = {
+ drill: 'r2-eviction', version: 1, mode: opts.execute ? 'execute' : 'dry-run', started_at: nowIso(), finished_at: null,
+ vod_id: null, object_id: null, key: null, http: !!opts.http, base_url: opts.baseUrl || null,
+ buckets: { b2: storage.bucketFor('b2'), r2: storage.bucketFor('r2') },
+ checks: {}, steps: [], locations: {}, verdict: null, reason: null,
+ };
+ const step = (name, ok, detail = {}) => { art.steps.push({ name, ok, at: nowIso(), ...detail }); return ok; };
+ const locations = () => (art.object_id ? model.listLocations(art.object_id).map(l => ({ provider: l.provider, state: l.state, size_bytes: l.size_bytes, verified_at: l.verified_at })) : []);
+ const refuse = (reason) => { art.verdict = 'refused'; art.reason = reason; art.finished_at = nowIso(); return art; };
+ const fail = (reason) => { art.verdict = 'fail'; art.reason = reason; art.finished_at = nowIso(); art.locations.at_failure = locations(); return art; };
+
+ if (!storage.providerConfigured('b2') || !storage.providerConfigured('r2')) return refuse('B2 and R2 must both be configured (MEDIA_B2_*, MEDIA_R2_*)');
+
+ // ── Choose the VOD ──
+ let vod = null;
+ if (opts.vodId) vod = db.get('SELECT * FROM vods WHERE id = ?', [opts.vodId]);
+ else if (opts.pick) {
+ const cands = db.all(`SELECT * FROM vods WHERE storage_provider = 'r2' AND COALESCE(is_recording, 0) = 0 AND COALESCE(clips_only, 0) = 0
+ AND file_path IS NOT NULL AND COALESCE(file_size, 0) > 0 AND file_size <= ?
+ ${opts.http ? "AND COALESCE(visibility, CASE WHEN is_public THEN 'public' ELSE 'private' END) != 'private'" : ''}
+ ORDER BY file_size ASC LIMIT 50`, [Math.max(1, opts.maxMb) * MIB]);
+ vod = cands.find(v => !model.isHeldRow(v)) || null;
+ if (!vod) return refuse(`no VOD served from R2 fits (not recording, not held${opts.http ? ', public or unlisted' : ''}, at most ${opts.maxMb} MB)`);
+ } else return refuse('name a VOD (--vod <id>) or --pick one');
+ if (!vod) return refuse(`vod ${opts.vodId} not found`);
+ art.vod_id = vod.id;
+ art.object_id = vod.object_id || null;
+ art.key = storage.keyForVod(vod);
+ const vis = vod.visibility || (vod.is_public ? 'public' : 'private');
+ art.checks.vod = { app_id: vod.app_id, storage_provider: storage.providerOf(vod), is_recording: !!vod.is_recording, visibility: vis, file_size: Number(vod.file_size) || null };
+ if (storage.providerOf(vod) !== 'r2') return refuse(`vod ${vod.id} is served from ${storage.providerOf(vod)}, not R2: there is no R2 copy to evict`);
+ if (vod.is_recording) return refuse(`vod ${vod.id} is recording`);
+ if (vod.clips_only) return refuse(`vod ${vod.id} is a clips-only recording`);
+ if (model.isHeldRow(vod)) return refuse(`vod ${vod.id} is under a retention hold (holds freeze placement)`);
+ if (opts.http && vis === 'private') return refuse(`vod ${vod.id} is private: /v/<id> would not serve it anonymously (use --no-http)`);
+ if (opts.http && !opts.baseUrl) return refuse('--base-url (or MEDIA_PUBLIC_URL) is needed for the HTTP checks');
+ art.locations.before = locations();
+
+ // ── Both copies agree (size, and the first MiB) ──
+ const [b2, r2] = await Promise.all([storage.headObject('b2', art.key), storage.headObject('r2', art.key)]);
+ art.checks.head_before = { b2, r2 };
+ if (!b2) return refuse('the B2 canonical copy is missing: evicting R2 would lose the only copy');
+ if (!r2) return refuse('there is no R2 copy (HEAD 404) although the row says r2');
+ if (b2.size !== r2.size) return refuse(`B2 (${b2.size}) and R2 (${r2.size}) differ in size: the canonical copy may be corrupt`);
+ if (vod.file_size && Number(vod.file_size) !== b2.size) return refuse(`the copies (${b2.size}) differ from the recorded size (${vod.file_size})`);
+ const b2Loc = (art.locations.before || []).find(l => l.provider === 'b2');
+ if (b2Loc && ['corrupt', 'missing'].includes(b2Loc.state)) return refuse(`media_locations marks the B2 copy ${b2Loc.state}`);
+ const [shaB2, shaR2] = await Promise.all([
+ firstMibSha(await storage.presignGet('b2', art.key, 600), fetchImpl),
+ firstMibSha(await storage.presignGet('r2', art.key, 600), fetchImpl),
+ ]);
+ art.checks.first_mib = { b2: shaB2.sha256, r2: shaR2.sha256 };
+ if (shaB2.sha256 !== shaR2.sha256) return refuse('B2 and R2 differ in their first MiB: the canonical copy may be corrupt');
+ const expectSha = shaR2.sha256;
+
+ /** Where playback goes now: GET /v/<id>?raw=1 (or resolvePlayback in-process) and the first MiB served there. */
+ const served = async () => {
+ if (!opts.http) {
+ const plan = await storage.resolvePlayback(db.get('SELECT * FROM vods WHERE id = ?', [vod.id]));
+ if (!plan || plan.kind !== 'redirect') return { provider: plan ? plan.kind : null };
+ const got = await firstMibSha(plan.url, fetchImpl);
+ return { via: 'resolvePlayback', provider: plan.provider, status: 302, first_mib_sha256: got.sha256 };
+ }
+ const url = `${String(opts.baseUrl).replace(/\/+$/, '')}/v/${vod.id}?raw=1`;
+ const res = await fetchImpl(url, { redirect: 'manual', headers: { Accept: 'video/*' } });
+ const location = res.headers.get('location');
+ if (res.status !== 302 || !location) return { via: 'http', status: res.status, provider: null };
+ const got = await firstMibSha(location, fetchImpl);
+ return { via: 'http', url, status: 302, provider: providerOfUrl(location, storage), host: new URL(location).host, range_status: got.status, first_mib_sha256: got.sha256 };
+ };
+
+ if (!opts.execute) {
+ const now = opts.http ? null : await served().catch(e => ({ error: e.message }));
+ step('plan', true, {
+ would: ['evict: demoteFromR2 (B2 canonical checked, R2 copy deleted, row -> b2)', 'prove: /v/<id> redirects to B2 and serves the same first MiB',
+ 're-warm: promoteToR2 (B2 copied to R2, row -> r2)', 'prove: /v/<id> redirects to R2 again'],
+ served_now: opts.http ? 'not requested in a dry run (GET /v counts a view)' : now,
+ size_bytes: b2.size,
+ });
+ art.verdict = 'dry-run';
+ art.finished_at = nowIso();
+ return art;
+ }
+
+ // 1. before
+ const before = await served();
+ if (!step('before: served from R2', before.provider === 'r2' && before.first_mib_sha256 === expectSha, before)) return fail('playback is not coming from R2 before the eviction');
+ // 2. evict
+ const ev = await storage.demoteFromR2(vod.id);
+ const rowAfterEvict = db.get('SELECT storage_provider FROM vods WHERE id = ?', [vod.id]);
+ const r2Gone = await storage.headObject('r2', art.key);
+ art.locations.after_evict = locations();
+ if (!step('evict: R2 copy removed', !!(ev && ev.ok) && !r2Gone && rowAfterEvict.storage_provider === 'b2', { result: ev, r2_head: r2Gone, storage_provider: rowAfterEvict.storage_provider })) {
+ return fail('the eviction did not complete');
+ }
+ // 3. served from B2
+ const fromB2 = await served();
+ const b2Head = await storage.headObject('b2', art.key);
+ if (!step('from B2: served from the canonical copy', fromB2.provider === 'b2' && fromB2.first_mib_sha256 === expectSha && !!b2Head && b2Head.size === b2.size, { ...fromB2, b2_head: b2Head })) {
+ return fail('after the eviction the VOD is not served from B2');
+ }
+ // 4. re-warm
+ const pr = await storage.promoteToR2(vod.id);
+ const r2Back = await storage.headObject('r2', art.key);
+ const rowAfterWarm = db.get('SELECT storage_provider FROM vods WHERE id = ?', [vod.id]);
+ if (!step('re-warm: R2 copy restored from B2', !!(pr && pr.ok) && !!r2Back && r2Back.size === b2.size && rowAfterWarm.storage_provider === 'r2', { result: pr, r2_head: r2Back, storage_provider: rowAfterWarm.storage_provider })) {
+ return fail('the re-warm did not complete (the VOD is still served from B2)');
+ }
+ // 5. after
+ const after = await served();
+ art.locations.after = locations();
+ const locOk = !art.object_id || ['b2', 'r2'].every(p => (art.locations.after.find(l => l.provider === p) || {}).state === 'present');
+ if (!step('after: served from R2 again', after.provider === 'r2' && after.first_mib_sha256 === expectSha && locOk, { ...after, locations_present: locOk })) {
+ return fail('after the re-warm the VOD is not served from R2');
+ }
+ art.verdict = 'pass';
+ art.finished_at = nowIso();
+ return art;
+}
+
+function summarize(art) {
+ const lines = [`R2 eviction drill (${art.mode}) vod ${art.vod_id ?? '-'} ${art.key || ''}: ${art.verdict}${art.reason ? ` — ${art.reason}` : ''}`];
+ for (const s of art.steps) lines.push(` ${s.ok ? 'ok ' : 'FAIL'} ${s.name}`);
+ return lines.join('\n');
+}
+
+if (require.main === module) {
+ const opts = parseArgs(process.argv);
+ if (opts.db) process.env.DB_PATH = path.resolve(opts.db);
+ const config = require('../server/config');
+ if (!opts.baseUrl) opts.baseUrl = config.publicUrl;
+ const storage = require('../server/vod/vod-storage');
+ const db = require('../server/db/database');
+ const model = require('../server/objects/model');
+ (async () => {
+ const art = await runDrill(opts, { storage, db, model });
+ const out = opts.out || path.join(path.dirname(config.db.path), 'drills', `r2-eviction-${art.vod_id ?? 'none'}-${art.started_at.replace(/[:.]/g, '-')}.json`);
+ fs.mkdirSync(path.dirname(out), { recursive: true });
+ fs.writeFileSync(out, JSON.stringify(art, null, 2));
+ console.log(opts.json ? JSON.stringify(art, null, 2) : `${summarize(art)}\nartifact: ${out}`);
+ db.close();
+ process.exit(art.verdict === 'pass' || art.verdict === 'dry-run' ? 0 : art.verdict === 'refused' ? 2 : 1);
+ })().catch((err) => { console.error(err); process.exit(1); });
+}
+
+module.exports = { runDrill, providerOfUrl, summarize, parseArgs };
@@ -342,7 +342,7 @@ function tenantCors(req, res, next) {
@@ -342,7 +342,7 @@ function tenantCors(req, res, next) {
if (origin && req.appRow && db.appAllowedOrigins(req.appRow).includes(origin)) {
if (origin && req.appRow && db.appAllowedOrigins(req.appRow).includes(origin)) {
res.set('Access-Control-Allow-Origin', origin);
res.set('Access-Control-Allow-Origin', origin);
res.set('Vary', 'Origin');
res.set('Vary', 'Origin');
- res.set('Access-Control-Allow-Headers', 'Authorization, Content-Type');
+ res.set('Access-Control-Allow-Headers', 'Authorization, Content-Type, X-Upload-Token, X-Content-SHA256, Idempotency-Key');
res.set('Access-Control-Allow-Methods', 'GET, POST, PUT, DELETE, OPTIONS');
res.set('Access-Control-Allow-Methods', 'GET, POST, PUT, DELETE, OPTIONS');
}
}
next();
next();
@@ -55,6 +55,27 @@ const config = {
@@ -55,6 +55,27 @@ const config = {
publicMaxMb: intEnv('MEDIA_PUBLIC_OBJECT_MAX_MB', 500),
publicMaxMb: intEnv('MEDIA_PUBLIC_OBJECT_MAX_MB', 500),
publicTargetMb: intEnv('MEDIA_PUBLIC_OBJECT_TARGET_MB', 256),
publicTargetMb: intEnv('MEDIA_PUBLIC_OBJECT_TARGET_MB', 256),
publicWarnMb: intEnv('MEDIA_PUBLIC_OBJECT_WARN_MB', 384),
publicWarnMb: intEnv('MEDIA_PUBLIC_OBJECT_WARN_MB', 384),
+ // Multipart uploads (server/objects/multipart.js): parts are stored locally and assembled at complete.
+ multipartMaxMb: intEnv('MEDIA_MULTIPART_MAX_MB', 20480), // largest object a multipart upload may declare
+ multipartMinPartMb: intEnv('MEDIA_MULTIPART_MIN_PART_MB', 5), // every part but the last is at least this
+ multipartMaxPartMb: intEnv('MEDIA_MULTIPART_MAX_PART_MB', 256),
+ multipartDefaultPartMb: intEnv('MEDIA_MULTIPART_PART_MB', 64),
+ multipartTtlHours: intEnv('MEDIA_MULTIPART_TTL_HOURS', 24), // an unfinished session's parts are purged after this
+ uploadMinFreeMb: intEnv('MEDIA_UPLOAD_MIN_FREE_MB', 10240), // multipart needs 2x its size free plus this
+ },
+
+ // Job system (server/jobs/; docs/object-model.md#jobs).
+ jobs: {
+ enabled: !['0', 'false', 'off'].includes(String(process.env.MEDIA_JOBS_ENABLED || '').toLowerCase()),
+ pollMs: intEnv('MEDIA_JOBS_POLL_MS', 5000),
+ lightConcurrency: Math.max(1, intEnv('MEDIA_JOBS_LIGHT_CONCURRENCY', 2)), // thumbnails, scans
+ heavyConcurrency: Math.max(1, intEnv('MEDIA_JOBS_HEAVY_CONCURRENCY', 1)), // split, remux
+ // Heavy jobs wait while a recording is running (as the health job does), unless this is on.
+ heavyWhileRecording: ['1', 'true', 'on'].includes(String(process.env.MEDIA_JOBS_HEAVY_WHILE_RECORDING || '').toLowerCase()),
+ leaseS: Math.max(30, intEnv('MEDIA_JOBS_LEASE_S', 120)),
+ // The size-invariant validator runs this often (hours; 0 = only on demand). It proposes jobs, never runs them.
+ invariantScanHours: intEnv('MEDIA_INVARIANT_SCAN_HOURS', 24),
+ retentionDays: intEnv('MEDIA_JOBS_RETENTION_DAYS', 30), // finished thumbnail jobs are pruned after this
},
},
// Scheduled copy verification (server/objects/verify-job.js; docs/object-model.md#scheduled-verification).
// Scheduled copy verification (server/objects/verify-job.js; docs/object-model.md#scheduled-verification).
@@ -28,6 +28,7 @@ function getDb() {
@@ -28,6 +28,7 @@ function getDb() {
const schema = fs.readFileSync(path.join(__dirname, 'schema.sql'), 'utf8');
const schema = fs.readFileSync(path.join(__dirname, 'schema.sql'), 'utf8');
database.exec(schema);
database.exec(schema);
migrateColumns();
migrateColumns();
+ migrateJobsTable(schema);
ensureObjectTriggers();
ensureObjectTriggers();
normalizeThumbnailUrls();
normalizeThumbnailUrls();
seedSettings();
seedSettings();
@@ -89,6 +90,35 @@ function migrateColumns() {
@@ -89,6 +90,35 @@ function migrateColumns() {
}
}
}
}
+/**
+ * media_jobs was created schema-only (integer id, no app_id, status 'done') before the job worker
+ * existed; nothing ever wrote it. Rebuild that shape into the current one from schema.sql, keeping
+ * any rows (id mjob_legacy_<n>, tenant from the object, done -> succeeded), then create the indexes
+ * that need the new columns. Idempotent: a database already in the new shape only gets its indexes.
+ */
+function migrateJobsTable(schema) {
+ const cols = database.prepare('PRAGMA table_info(media_jobs)').all().map(c => c.name);
+ if (!cols.includes('app_id')) {
+ const ddl = /CREATE TABLE IF NOT EXISTS media_jobs \([\s\S]*?\n\);/.exec(schema);
+ if (!ddl) throw new Error('schema.sql has no media_jobs table');
+ database.transaction(() => {
+ database.exec('ALTER TABLE media_jobs RENAME TO media_jobs_v0');
+ database.exec(ddl[0]);
+ const moved = database.prepare(`INSERT INTO media_jobs (id, app_id, object_id, job_type, status, attempts, checkpoint, error, created_at, updated_at)
+ SELECT 'mjob_legacy_' || v.id, COALESCE((SELECT o.app_id FROM media_objects o WHERE o.id = v.object_id), 'unknown'), v.object_id, v.job_type,
+ CASE v.status WHEN 'done' THEN 'succeeded' ELSE v.status END, v.attempts, v.checkpoint, v.error, v.created_at, v.updated_at
+ FROM media_jobs_v0 v`).run().changes;
+ database.exec('DROP TABLE media_jobs_v0');
+ console.log(`[DB] Rebuilt media_jobs for the job worker (${moved} row(s) kept)`);
+ })();
+ }
+ database.exec(`
+ CREATE INDEX IF NOT EXISTS idx_media_jobs_status ON media_jobs(status, job_type);
+ CREATE INDEX IF NOT EXISTS idx_media_jobs_app ON media_jobs(app_id, id);
+ CREATE INDEX IF NOT EXISTS idx_media_jobs_object ON media_jobs(object_id, job_type, status);
+ CREATE UNIQUE INDEX IF NOT EXISTS idx_media_jobs_idem ON media_jobs(app_id, idempotency_key) WHERE idempotency_key IS NOT NULL;`);
+}
+
/**
/**
* Object-model guards that must hold on EVERY delete path, including the many
* Object-model guards that must hold on EVERY delete path, including the many
* inherited ones that DELETE a vods/clips/files/pastes row directly:
* inherited ones that DELETE a vods/clips/files/pastes row directly:
@@ -275,20 +275,65 @@ CREATE TABLE IF NOT EXISTS media_variants (
@@ -275,20 +275,65 @@ CREATE TABLE IF NOT EXISTS media_variants (
UNIQUE(object_id, variant_name)
UNIQUE(object_id, variant_name)
);
);
+-- Generic derivative/maintenance jobs (server/jobs/; docs/object-model.md#jobs). A database made before
+-- the worker existed has the schema-only shape (integer id, no app_id); database.js migrateJobsTable()
+-- rebuilds it into this one. The other indexes are created there too, after the rebuild.
CREATE TABLE IF NOT EXISTS media_jobs (
CREATE TABLE IF NOT EXISTS media_jobs (
- id INTEGER PRIMARY KEY AUTOINCREMENT,
- object_id TEXT,
- job_type TEXT NOT NULL,
- status TEXT NOT NULL DEFAULT 'queued' CHECK(status IN ('queued', 'running', 'done', 'failed', 'cancelled')),
+ id TEXT PRIMARY KEY, -- mjob_<ULID>
+ app_id TEXT NOT NULL, -- the tenant that owns the job
+ object_id TEXT, -- the object it works on (NULL for tenant-wide jobs: invariant.scan)
+ job_type TEXT NOT NULL, -- thumbnail.regenerate | invariant.scan | object.split | object.remux
+ status TEXT NOT NULL DEFAULT 'queued' CHECK(status IN ('proposed', 'queued', 'running', 'succeeded', 'failed', 'cancelled')),
+ idempotency_key TEXT, -- unique per tenant: a repeat answers with the same job
+ request_hash TEXT, -- sha256 of { type, object_id, params }: a different request under a used key is refused
+ params TEXT NOT NULL DEFAULT '{}',
+ result TEXT,
attempts INTEGER NOT NULL DEFAULT 0,
attempts INTEGER NOT NULL DEFAULT 0,
- checkpoint TEXT,
+ max_attempts INTEGER NOT NULL DEFAULT 3,
+ run_after DATETIME, -- queued: not before (retry backoff)
+ lease_until DATETIME, -- running: renewed while the worker holds the job
+ checkpoint TEXT, -- handler progress (JSON); a retry resumes from it
error TEXT,
error TEXT,
+ error_code TEXT, -- stable code of the last failure (media_unavailable, quota_exceeded, …)
+ cancel_requested INTEGER NOT NULL DEFAULT 0,
+ created_by TEXT, -- svc:<id> | app:<app> | app:<app>:user:<id> | system:<what>
+ owner_user_id INTEGER, -- the app's user it was created for (X-OV-User-Id), if any
+ decided_by TEXT, -- who approved or rejected a proposal
+ decided_at DATETIME,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
- updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
+ updated_at DATETIME DEFAULT CURRENT_TIMESTAMP,
+ started_at DATETIME,
+ finished_at DATETIME
);
);
CREATE INDEX IF NOT EXISTS idx_media_jobs_status ON media_jobs(status, job_type);
CREATE INDEX IF NOT EXISTS idx_media_jobs_status ON media_jobs(status, job_type);
+-- Multipart uploads of native v2 objects (server/objects/multipart.js). Parts are stored under
+-- OBJECTS_PATH/.parts/<upload id>/ and assembled at complete; one active session per object.
+CREATE TABLE IF NOT EXISTS media_uploads (
+ id TEXT PRIMARY KEY, -- mup_<ULID>
+ object_id TEXT NOT NULL,
+ app_id TEXT NOT NULL,
+ part_size INTEGER NOT NULL,
+ total_size INTEGER NOT NULL,
+ parts_expected INTEGER NOT NULL,
+ status TEXT NOT NULL DEFAULT 'active' CHECK(status IN ('active', 'completing', 'completed', 'aborted', 'expired')),
+ created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
+ updated_at DATETIME DEFAULT CURRENT_TIMESTAMP,
+ expires_at DATETIME NOT NULL,
+ completed_at DATETIME
+);
+CREATE INDEX IF NOT EXISTS idx_media_uploads_object ON media_uploads(object_id, status);
+CREATE INDEX IF NOT EXISTS idx_media_uploads_expiry ON media_uploads(status, expires_at);
+
+CREATE TABLE IF NOT EXISTS media_upload_parts (
+ upload_id TEXT NOT NULL,
+ part_number INTEGER NOT NULL, -- 1-based
+ size_bytes INTEGER NOT NULL,
+ sha256 TEXT NOT NULL,
+ received_at DATETIME DEFAULT CURRENT_TIMESTAMP,
+ PRIMARY KEY (upload_id, part_number)
+);
+
-- Retention holds: an object with an unreleased hold cannot be deleted or moved between tiers.
-- Retention holds: an object with an unreleased hold cannot be deleted or moved between tiers.
CREATE TABLE IF NOT EXISTS media_holds (
CREATE TABLE IF NOT EXISTS media_holds (
id INTEGER PRIMARY KEY AUTOINCREMENT,
id INTEGER PRIMARY KEY AUTOINCREMENT,
@@ -9,6 +9,8 @@
@@ -9,6 +9,8 @@
* clip.ready | clip.failed → media.clip.ready | media.clip.failed (subject clip <id>)
* clip.ready | clip.failed → media.clip.ready | media.clip.failed (subject clip <id>)
* media.object.uploaded → media.object.uploaded (subject object <id>)
* media.object.uploaded → media.object.uploaded (subject object <id>)
* storage.alert | storage.recovered → media.storage.alert | media.storage.recovered
* storage.alert | storage.recovered → media.storage.alert | media.storage.recovered
+ * job state changes (server/jobs/queue.js) → media.job.proposed | queued | started | retrying |
+ * succeeded | failed | cancelled (subject job <id>; Events only, no webhook)
*
*
* The outbox row is written INSIDE the SQLite transaction that makes the state change it describes
* The outbox row is written INSIDE the SQLite transaction that makes the state change it describes
* (record(), called from webhooks.announce()): the event exists if and only if the change committed,
* (record(), called from webhooks.announce()): the event exists if and only if the change committed,
@@ -36,6 +38,12 @@ const TYPES = {
@@ -36,6 +38,12 @@ const TYPES = {
'storage.recovered': ['media.storage.recovered', 'storage'],
'storage.recovered': ['media.storage.recovered', 'storage'],
};
};
+// media.job.<transition>: progress is low priority, the outcome (and a proposal waiting for its owner) important.
+const JOB_TRANSITIONS = {
+ proposed: 'important', queued: 'low', started: 'low', retrying: 'low',
+ succeeded: 'important', failed: 'important', cancelled: 'important',
+};
+
let outbox = null;
let outbox = null;
const stats = { queued: 0, lastError: null };
const stats = { queued: 0, lastError: null };
@@ -80,10 +88,6 @@ function slim(appId, data) {
@@ -80,10 +88,6 @@ function slim(appId, data) {
* or the webhook event has no durable twin.
* or the webhook event has no durable twin.
*/
*/
function record(webhookEvent, appId, data) {
function record(webhookEvent, appId, data) {
- if (!outbox) return null;
- // Developer-project sandbox tenants (ADR-014) produce no platform events: sandbox activity must
- // never reach production consumers.
- if (appId && db.isSandboxTenant(appId)) return null;
const map = TYPES[webhookEvent];
const map = TYPES[webhookEvent];
if (!map) return null;
if (!map) return null;
const [eventType, subjectType] = map;
const [eventType, subjectType] = map;
@@ -91,18 +95,51 @@ function record(webhookEvent, appId, data) {
@@ -91,18 +95,51 @@ function record(webhookEvent, appId, data) {
const subject = subjectType === 'storage'
const subject = subjectType === 'storage'
? { type: 'storage', id: String((data && data.kind) || 'storage') }
? { type: 'storage', id: String((data && data.kind) || 'storage') }
: { type: subjectType, id: String(id == null ? 'unknown' : id) };
: { type: subjectType, id: String(id == null ? 'unknown' : id) };
+ return enqueue(eventType, appId, subject, data, eventType === 'media.storage.recovered' ? 'low' : 'important');
+}
+
+/**
+ * Queue media.job.<transition> for a job row (queue.jobPublic shape). Same rule as record(): call it
+ * inside the transaction that changes the job's state; it throws when the insert fails.
+ */
+function recordJob(transition, job) {
+ const priority = JOB_TRANSITIONS[transition];
+ if (!priority) throw new Error(`unknown job transition ${transition}`);
+ return enqueue(`media.job.${transition}`, job.app_id, { type: 'job', id: String(job.id) }, job, priority);
+}
+
+function enqueue(eventType, appId, subject, data, priority) {
+ if (!outbox) return null;
+ // Developer-project sandbox tenants (ADR-014) produce no platform events: sandbox activity must
+ // never reach production consumers.
+ if (appId && db.isSandboxTenant(appId)) return null;
const env = outbox.enqueue({
const env = outbox.enqueue({
event_type: eventType,
event_type: eventType,
actor: { type: 'service', id: 'media' },
actor: { type: 'service', id: 'media' },
subject,
subject,
visibility: 'internal',
visibility: 'internal',
- priority: eventType === 'media.storage.recovered' ? 'low' : 'important',
+ priority,
payload: slim(appId, data),
payload: slim(appId, data),
});
});
stats.queued++;
stats.queued++;
return env;
return env;
}
}
+/**
+ * Operator scripts (scripts/media-jobs.js) change job state in the same database as the running
+ * service. When the service's outbox table exists (the outbox is on there), the script writes its
+ * events into it too, and the service's relay publishes them. No relay runs in the script.
+ */
+function initWriter() {
+ if (outbox) return outbox;
+ const raw = db.getDb();
+ if (!raw.prepare("SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = 'event_outbox'").get()) return null;
+ const client = createClient({ baseUrls: { events: 'http://127.0.0.1:9' }, retries: 0 });
+ outbox = createOutbox(raw, { events: createEventsClient(client, { source: 'media' }) });
+ outbox.ensureSchema();
+ return outbox;
+}
+
/** Wake the relay once the transaction that queued events has committed. */
/** Wake the relay once the transaction that queued events has committed. */
function kick() {
function kick() {
if (outbox) setImmediate(() => outbox && outbox.kick());
if (outbox) setImmediate(() => outbox && outbox.kick());
@@ -115,4 +152,4 @@ function status() {
@@ -115,4 +152,4 @@ function status() {
function _reset() { if (outbox) outbox.stop(); outbox = null; stats.queued = 0; stats.lastError = null; }
function _reset() { if (outbox) outbox.stop(); outbox = null; stats.queued = 0; stats.lastError = null; }
-module.exports = { init, record, kick, status, TYPES, _reset };
+module.exports = { init, initWriter, record, recordJob, kick, status, TYPES, JOB_TRANSITIONS, _reset };
@@ -33,6 +33,7 @@ const observability = require('./observability');
@@ -33,6 +33,7 @@ const observability = require('./observability');
const instrumented = observability.instrument(app, { release: release.release });
const instrumented = observability.instrument(app, { release: release.release });
// Object API v2 content uploads read the raw request body, so they sit ahead of the body parsers.
// Object API v2 content uploads read the raw request body, so they sit ahead of the body parsers.
app.put('/api/v2/:app/objects/:id/content', ...require('./objects/routes').contentHandlers);
app.put('/api/v2/:app/objects/:id/content', ...require('./objects/routes').contentHandlers);
+app.put('/api/v2/:app/objects/:id/multipart/:uploadId/parts/:n', ...require('./objects/routes').partHandlers);
app.use(express.json({ limit: '2mb' }));
app.use(express.json({ limit: '2mb' }));
// ── Visitor sign-in (OAuth client `media` on the Network; same module as Community/Tools) ──
// ── Visitor sign-in (OAuth client `media` on the Network; same module as Community/Tools) ──
@@ -78,7 +79,7 @@ app.options(['/api/v1/:app/*', '/api/v2/:app/*'], (req, res) => {
@@ -78,7 +79,7 @@ app.options(['/api/v1/:app/*', '/api/v2/:app/*'], (req, res) => {
if (origin && appRow && db.appAllowedOrigins(appRow).includes(origin)) {
if (origin && appRow && db.appAllowedOrigins(appRow).includes(origin)) {
res.set('Access-Control-Allow-Origin', origin);
res.set('Access-Control-Allow-Origin', origin);
res.set('Vary', 'Origin');
res.set('Vary', 'Origin');
- res.set('Access-Control-Allow-Headers', 'Authorization, Content-Type');
+ res.set('Access-Control-Allow-Headers', 'Authorization, Content-Type, X-Upload-Token, X-Content-SHA256, Idempotency-Key');
res.set('Access-Control-Allow-Methods', 'GET, POST, PUT, DELETE, OPTIONS');
res.set('Access-Control-Allow-Methods', 'GET, POST, PUT, DELETE, OPTIONS');
}
}
res.sendStatus(204);
res.sendStatus(204);
@@ -143,6 +144,7 @@ app.use('/api/v1/:app/thumbnails', require('./thumbnails/routes'));
@@ -143,6 +144,7 @@ app.use('/api/v1/:app/thumbnails', require('./thumbnails/routes'));
app.use('/api/v1/:app/assets', require('./assets/routes'));
app.use('/api/v1/:app/assets', require('./assets/routes'));
app.use('/api/v1/:app/admin/storage', require('./admin/routes'));
app.use('/api/v1/:app/admin/storage', require('./admin/routes'));
app.use('/api/v2/:app/objects', require('./objects/routes')); // canonical object API (docs/object-model.md)
app.use('/api/v2/:app/objects', require('./objects/routes')); // canonical object API (docs/object-model.md)
+app.use('/api/v2/:app/jobs', require('./jobs/routes')); // media jobs: thumbnails, split/remux, invariant scans
app.use('/o', require('./objects/routes').publicRouter); // object bytes (public, or signed)
app.use('/o', require('./objects/routes').publicRouter); // object bytes (public, or signed)
app.use('/', require('./public/routes')); // /v /c /p /t /f
app.use('/', require('./public/routes')); // /v /c /p /t /f
@@ -173,6 +175,7 @@ vodStorage.start(); // tiering sweep
@@ -173,6 +175,7 @@ vodStorage.start(); // tiering sweep
healthJob.start(); // health scan + quarantine cleanup
healthJob.start(); // health scan + quarantine cleanup
require('./vod/clip-jobs').start(); // failed clip re-cuts (auto-retry, hot-fetch)
require('./vod/clip-jobs').start(); // failed clip re-cuts (auto-retry, hot-fetch)
require('./objects/verify-job').start(); // scheduled copy verification (bounded batches; never deletes)
require('./objects/verify-job').start(); // scheduled copy verification (bounded batches; never deletes)
+require('./jobs/worker').start(); // media_jobs worker (light + heavy lanes; proposals wait for their owner)
// Descriptor watchdog: a leak here once pinned 80 GB of deleted recordings to the disk.
// Descriptor watchdog: a leak here once pinned 80 GB of deleted recordings to the disk.
every(5 * 60 * 1000, () => {
every(5 * 60 * 1000, () => {
try {
try {
@@ -185,6 +188,8 @@ every(60 * 60 * 1000, () => thumbService.cleanupOldThumbnails()); // stale live
@@ -185,6 +188,8 @@ every(60 * 60 * 1000, () => thumbService.cleanupOldThumbnails()); // stale live
// Object model: purge native objects whose soft-delete retention has passed (held ones are kept).
// Object model: purge native objects whose soft-delete retention has passed (held ones are kept).
every(60 * 60 * 1000, () => {
every(60 * 60 * 1000, () => {
try { const n = require('./objects/model').purgeExpired(); if (n) console.log(`[Objects] Purged ${n} expired deleted object(s)`); } catch (err) { console.warn('[Objects] purge:', err.message); }
try { const n = require('./objects/model').purgeExpired(); if (n) console.log(`[Objects] Purged ${n} expired deleted object(s)`); } catch (err) { console.warn('[Objects] purge:', err.message); }
+ // Incomplete multipart uploads past MEDIA_MULTIPART_TTL_HOURS: their parts are deleted (the objects stay uploading).
+ try { const r = require('./objects/multipart').purgeExpired(); if (r.expired || r.orphan_dirs) console.log(`[Objects] Multipart: ${r.expired} expired session(s), ${r.orphan_dirs} orphan part dir(s) removed`); } catch (err) { console.warn('[Objects] multipart purge:', err.message); }
});
});
// Project rows that have no media_object yet (first boot after the upgrade: all of them).
// Project rows that have no media_object yet (first boot after the upgrade: all of them).
setTimeout(() => {
setTimeout(() => {
@@ -229,6 +234,7 @@ function shutdown(signal) {
@@ -229,6 +234,7 @@ function shutdown(signal) {
try { vodStorage.stop(); } catch { /* */ }
try { vodStorage.stop(); } catch { /* */ }
try { healthJob.stop(); } catch { /* */ }
try { healthJob.stop(); } catch { /* */ }
try { require('./objects/verify-job').stop(); } catch { /* */ }
try { require('./objects/verify-job').stop(); } catch { /* */ }
+ try { require('./jobs/worker').stop(); } catch { /* */ }
try { auth.stopJwksRefresh(); } catch { /* */ }
try { auth.stopJwksRefresh(); } catch { /* */ }
try { require('./events')._reset(); } catch { /* */ }
try { require('./events')._reset(); } catch { /* */ }
for (const t of timers) clearInterval(t);
for (const t of timers) clearInterval(t);
new file mode 100644
new file mode 100644
@@ -0,0 +1,310 @@
@@ -0,0 +1,310 @@
+/**
+ * OpenVibe.Media — job types object.split and object.remux (heavy lane)
+ *
+ * Both make NEW objects from a ready vod/clip (or a native video/audio object) with a stream copy
+ * (ffmpeg -c copy: no re-encode). They never touch the source: its bytes, locations, visibility and
+ * lifecycle stay as they are. Each output is a native object in the same tenant with the source's
+ * owner, visibility `private` (the owner decides what becomes public), a verified local copy, and a
+ * `derived_from` relationship to the source ({ job_id, method, part, parts, start_seconds, … }).
+ *
+ * object.split params { parts | segment_seconds, … } one object per part, cut at keyframes (a part
+ * can start up to one keyframe interval early, so neighbours may overlap slightly).
+ * Progress is checkpointed per part: a retry resumes after the last finished part.
+ * result { source_id, parts: [{ part, object_id, start_seconds, duration_seconds, size_bytes }],
+ * oversized_parts, method: 'split' }
+ * object.remux params {} one object: the whole source remuxed (seek index,
+ * duration; MP4 +faststart). Also the source's `remux` variant.
+ * result { source_id, object_id, size_bytes, duration_seconds, method: 'remux' }
+ *
+ * Refused before any work: a source that is not ready (permanent), a tenant quota the output would
+ * exceed (permanent), less free disk than the source size plus MEDIA_UPLOAD_MIN_FREE_MB (retried later).
+ */
+'use strict';
+
+const fs = require('fs');
+const path = require('path');
+const crypto = require('crypto');
+const { spawn } = require('child_process');
+const config = require('../config');
+const db = require('../db/database');
+const model = require('../objects/model');
+const { JobError } = require('./queue');
+
+const MB = 1024 * 1024;
+const MAX_PARTS = 1000;
+const MEDIA_KINDS = ['vod', 'clip'];
+
+function isMediaObject(obj) {
+ return !!obj && (MEDIA_KINDS.includes(obj.kind) || /^(video|audio)\//.test(String(obj.mime_type || '')));
+}
+
+function extFor(obj, sourcePath) {
+ const fromPath = path.extname(String(sourcePath || obj.canonical_key || '')).toLowerCase();
+ if (['.webm', '.mp4', '.mkv', '.mov', '.m4a', '.ogg', '.mp3'].includes(fromPath)) return fromPath === '.mov' ? '.mp4' : fromPath;
+ const m = String(obj.mime_type || '');
+ if (m === 'video/mp4' || m === 'audio/mp4') return '.mp4';
+ if (m === 'video/x-matroska') return '.mkv';
+ if (m === 'audio/mpeg') return '.mp3';
+ if (m === 'audio/ogg' || m === 'video/ogg') return '.ogg';
+ return '.webm';
+}
+
+function mimeFor(ext, obj) {
+ return { '.webm': 'video/webm', '.mp4': 'video/mp4', '.mkv': 'video/x-matroska', '.m4a': 'audio/mp4', '.ogg': 'video/ogg', '.mp3': 'audio/mpeg' }[ext]
+ || obj.mime_type || 'application/octet-stream';
+}
+
+function existing(p) {
+ try { return p && fs.statSync(p).isFile() ? p : null; } catch { return null; }
+}
+
+/**
+ * Something ffmpeg can read for this object: a local path, or a presigned R2/B2 URL.
+ * Projected vods/clips resolve the way playback does; native objects through their locations.
+ */
+async function resolveSource(obj) {
+ const vodStorage = require('../vod/vod-storage');
+ const ref = model.parseLegacyRef(obj.legacy_ref);
+ if (ref && (ref.kind === 'vod' || ref.kind === 'clip')) {
+ const row = db.get(`SELECT * FROM ${ref.kind === 'vod' ? 'vods' : 'clips'} WHERE id = ?`, [Number(ref.id)]);
+ if (!row || !row.file_path) return null;
+ const local = ref.kind === 'vod'
+ ? existing(vodStorage.localPathForVod(row)) || existing(row.file_path)
+ : existing(row.file_path) || existing(path.join(path.resolve(config.vod.clipsPath), path.basename(row.file_path)));
+ if (local) return { input: local, name: row.file_path };
+ const src = await vodStorage.resolveMediaSource(row);
+ return src ? { input: src.value, name: row.file_path, remote: src.kind === 'url' } : null;
+ }
+ const locs = model.listLocations(obj.id);
+ const local = locs.find(l => l.provider === 'local' && l.state !== 'missing' && existing(l.key));
+ if (local) return { input: local.key, name: obj.canonical_key || local.key };
+ for (const p of ['r2', 'b2']) {
+ const l = locs.find(x => x.provider === p && !['missing', 'corrupt'].includes(x.state));
+ if (!l || !vodStorage.providerConfigured(p)) continue;
+ const url = await vodStorage.presignGet(p, l.key, 6 * 3600).catch(() => null);
+ if (url) return { input: url, name: l.key, remote: true };
+ }
+ return null;
+}
+
+function freeBytes(dir) {
+ try { const st = fs.statfsSync(dir); return Number(st.bavail) * Number(st.bsize); } catch { return Infinity; }
+}
+
+function sha256File(p) {
+ return new Promise((resolve, reject) => {
+ const h = crypto.createHash('sha256');
+ const s = fs.createReadStream(p);
+ s.on('data', (c) => h.update(c));
+ s.on('error', reject);
+ s.on('end', () => resolve(h.digest('hex')));
+ });
+}
+
+/** ffmpeg with a deadline and the job's abort signal. Resolves { ok, code, stderr }. */
+function ffmpeg(args, { signal, timeoutMs }) {
+ return new Promise((resolve) => {
+ let proc;
+ try { proc = spawn('ffmpeg', args, { stdio: ['ignore', 'ignore', 'pipe'] }); } catch (err) { return resolve({ ok: false, code: -1, stderr: err.message }); }
+ let stderr = '';
+ let done = false;
+ const finish = (r) => { if (done) return; done = true; clearTimeout(t); if (signal) signal.removeEventListener('abort', onAbort); resolve(r); };
+ const kill = () => { try { proc.kill('SIGKILL'); } catch { /* gone */ } };
+ const onAbort = () => kill();
+ const t = setTimeout(() => { stderr += '\n[timed out]'; kill(); }, timeoutMs);
+ if (signal) { if (signal.aborted) kill(); else signal.addEventListener('abort', onAbort, { once: true }); }
+ proc.stderr.on('data', (d) => { stderr = (stderr + d).slice(-4000); });
+ proc.on('error', (err) => finish({ ok: false, code: -1, stderr: err.message }));
+ proc.on('close', (code) => finish({ ok: code === 0, code, stderr }));
+ });
+}
+
+function inputArgs(src) {
+ return src.remote ? ['-reconnect', '1', '-reconnect_streamed', '1', '-reconnect_delay_max', '10'] : [];
+}
+
+/** Budget for one stream copy of `bytes`: ~45 s per GB, at least 3 min, at most 2 h. */
+function budgetMs(bytes) {
+ return Math.min(2 * 3600 * 1000, Math.max(180000, Math.round((Number(bytes) || 0) / 1e9 * 45000) + 60000));
+}
+
+function probeDuration(p) {
+ return require('../vod/media-tools').probeVodInfo(p).then(i => i.duration || 0).catch(() => 0);
+}
+
+function loadSource(job) {
+ const obj = model.getObject(job.object_id);
+ if (!obj || obj.app_id !== job.app_id) throw new JobError('not_found', 'The source object no longer exists', { permanent: true });
+ if (obj.lifecycle_status !== 'ready') throw new JobError('not_ready', `The source object is ${obj.lifecycle_status}`, { permanent: true });
+ if (!isMediaObject(obj)) throw new JobError('media.job.invalid', 'Only vod/clip objects (or video/audio objects) can be split or remuxed', { permanent: true });
+ return obj;
+}
+
+function checkRoom(obj, needBytes) {
+ const reserve = config.objects.uploadMinFreeMb * MB;
+ const free = freeBytes(config.objects.path);
+ if (free < needBytes + reserve) {
+ throw new JobError('insufficient_disk', `Not enough free disk: need ${Math.ceil((needBytes + reserve) / MB)} MB, ${Math.floor(free / MB)} MB free`, { retryAfterS: 1800 });
+ }
+ const app = db.getApp(obj.app_id);
+ const quota = Number(app && app.quota_bytes) || 0;
+ if (quota && model.usedBytes(obj.app_id) + needBytes > quota) {
+ throw new JobError('quota_exceeded', 'The tenant storage quota would be exceeded', { permanent: true });
+ }
+}
+
+function workDir(jobId) {
+ const d = path.join(config.objects.path, '.jobs', jobId);
+ fs.mkdirSync(d, { recursive: true });
+ return d;
+}
+
+function cleanupWork(jobId) {
+ try { fs.rmSync(path.join(config.objects.path, '.jobs', jobId), { recursive: true, force: true }); } catch { /* best effort */ }
+}
+
+/**
+ * Register one finished output file as a private native object derived from `src`: create the row,
+ * move the bytes under OBJECTS_PATH/<app>/<id>, verified local location, relationship; `alsoInTx`
+ * (the job checkpoint) commits with it. Returns the new object id.
+ */
+async function adopt({ src, file, ext, job, relation, metadata, variant, saveCheckpoint, checkpointFor }) {
+ const size = fs.statSync(file).size;
+ const hash = await sha256File(file);
+ const md = model.parseJson(src.metadata, {});
+ const id = require('openvibe-contracts').ids.newId('media');
+ const dest = path.join(config.objects.path, src.app_id, id);
+ fs.mkdirSync(path.dirname(dest), { recursive: true });
+ fs.renameSync(file, dest);
+ try {
+ saveCheckpoint(checkpointFor(id, size), () => {
+ db.run(`INSERT INTO media_objects (id, app_id, namespace, kind, owner_subject, owner_app, owner_user_id, visibility, lifecycle_status,
+ mime_type, size_bytes, content_hash, canonical_provider, canonical_key, legacy_ref, metadata)
+ VALUES (?, ?, ?, ?, ?, ?, ?, 'private', 'ready', ?, ?, ?, 'local', ?, NULL, ?)`,
+ [id, src.app_id, src.namespace || src.app_id, src.kind, src.owner_subject || null, src.owner_app || src.app_id, src.owner_user_id ?? null,
+ mimeFor(ext, src), size, hash, dest,
+ JSON.stringify({ title: md.title || null, derived_from: src.id, job_id: job.id, filename: `${src.id}${metadata.part ? `-part${metadata.part}` : '-remux'}${ext}`, ...metadata })]);
+ model.upsertLocation(id, { provider: 'local', key: dest, state: 'present', size_bytes: size, checksum: hash, verified: true });
+ model.setRelationship(id, 'derived_from', src.id, { job_id: job.id, ...relation });
+ if (variant) model.setVariant(src.id, variant, id, `${job.type}@1`);
+ });
+ } catch (err) {
+ try { fs.unlinkSync(dest); } catch { /* not moved */ }
+ throw err;
+ }
+ return { id, size };
+}
+
+// ── object.split ─────────────────────────────────────────────
+
+function validateSplit({ obj, params }) {
+ if (!obj) throw new JobError('media.job.invalid', 'object.split needs object_id', { permanent: true });
+ if (!isMediaObject(obj)) throw new JobError('media.job.invalid', 'Only vod/clip objects (or video/audio objects) can be split', { permanent: true });
+ const p = { ...(params || {}) };
+ const parts = p.parts != null ? Number(p.parts) : null;
+ const seg = p.segment_seconds != null ? Number(p.segment_seconds) : null;
+ if (parts == null && seg == null) throw new JobError('media.job.invalid', 'object.split needs params.parts (2-1000) or params.segment_seconds', { permanent: true });
+ if (parts != null && (!Number.isInteger(parts) || parts < 2 || parts > MAX_PARTS)) throw new JobError('media.job.invalid', 'params.parts must be an integer from 2 to 1000', { permanent: true });
+ if (seg != null && (!Number.isFinite(seg) || seg < 1)) throw new JobError('media.job.invalid', 'params.segment_seconds must be at least 1', { permanent: true });
+ return p;
+}
+
+async function runSplit(job, ctx) {
+ const src = loadSource(job);
+ const source = await resolveSource(src);
+ if (!source) throw new JobError('media_unavailable', 'The source bytes are unavailable (no local file and no cloud copy)', { permanent: true });
+ const md = model.parseJson(src.metadata, {});
+ const duration = Number(md.duration_seconds) || await probeDuration(source.input);
+ if (!(duration > 0)) throw new JobError('duration_unknown', 'The source duration is unknown: remux it first (object.remux)', { permanent: true });
+ const parts = job.params.segment_seconds
+ ? Math.min(MAX_PARTS, Math.ceil(duration / Number(job.params.segment_seconds)))
+ : Math.min(MAX_PARTS, Number(job.params.parts));
+ const seg = job.params.segment_seconds ? Number(job.params.segment_seconds) : Math.ceil(duration / parts);
+ const cp = ctx.checkpoint && Array.isArray(ctx.checkpoint.parts) ? ctx.checkpoint : { parts: [] };
+ const doneParts = new Set(cp.parts.map(p => p.part));
+ const remaining = Math.max(0, (Number(src.size_bytes) || 0) - cp.parts.reduce((a, p) => a + p.size_bytes, 0));
+ checkRoom(src, remaining);
+ const ext = extFor(src, source.name);
+ const dir = workDir(job.id);
+ const maxBytes = Number(job.params.max_bytes) || require('../objects/invariant').thresholds().maxBytes;
+
+ for (let i = 0; i < parts; i++) {
+ const part = i + 1;
+ const start = i * seg;
+ if (start >= duration) break;
+ if (doneParts.has(part)) continue;
+ if (ctx.signal.aborted) throw ctx.signal.reason || new Error('aborted');
+ const len = Math.min(seg, duration - start);
+ const out = path.join(dir, `part-${part}${ext}`);
+ // A cloud source is presigned again for every part: a long split outlives one presigned URL.
+ const src2 = part === 1 || !source.remote ? source : (await resolveSource(src)) || source;
+ const r = await ffmpeg(['-y', '-nostdin', '-v', 'error', ...inputArgs(src2), '-ss', String(start), '-i', src2.input, '-t', String(len),
+ '-map', '0:v?', '-map', '0:a?', '-c', 'copy', '-avoid_negative_ts', 'make_zero', ...(ext === '.mp4' ? ['-movflags', '+faststart'] : []), out],
+ { signal: ctx.signal, timeoutMs: budgetMs((Number(src.size_bytes) || 0) / parts * 2) });
+ if (ctx.signal.aborted) throw ctx.signal.reason || new Error('aborted');
+ if (!r.ok || !existing(out) || fs.statSync(out).size === 0) {
+ try { fs.unlinkSync(out); } catch { /* none */ }
+ throw new JobError('ffmpeg_failed', `part ${part}: ffmpeg exited ${r.code}: ${String(r.stderr).trim().split('\n').slice(-2).join(' ').slice(0, 300)}`);
+ }
+ await adopt({
+ src, file: out, ext, job, variant: null,
+ relation: { method: 'split', part, parts, start_seconds: start, duration_seconds: len },
+ metadata: { part, parts, start_seconds: start, duration_seconds: len },
+ saveCheckpoint: ctx.saveCheckpoint,
+ checkpointFor: (id, size) => ({ parts: [...cp.parts, { part, object_id: id, start_seconds: start, duration_seconds: len, size_bytes: size }] }),
+ });
+ cp.parts = ctx.checkpoint.parts;
+ }
+ cleanupWork(job.id);
+ const list = [...cp.parts].sort((a, b) => a.part - b.part);
+ return {
+ source_id: src.id, method: 'split', parts: list,
+ oversized_parts: list.filter(p => p.size_bytes > maxBytes).map(p => p.part),
+ source_unchanged: true,
+ note: 'The parts are private objects. The source is unchanged: it stays as it is until its owner changes it.',
+ };
+}
+
+// ── object.remux ─────────────────────────────────────────────
+
+function validateRemux({ obj }) {
+ if (!obj) throw new JobError('media.job.invalid', 'object.remux needs object_id', { permanent: true });
+ if (!isMediaObject(obj)) throw new JobError('media.job.invalid', 'Only vod/clip objects (or video/audio objects) can be remuxed', { permanent: true });
+ return {};
+}
+
+async function runRemux(job, ctx) {
+ const src = loadSource(job);
+ if (ctx.checkpoint && ctx.checkpoint.object_id) { // finished before an interruption
+ cleanupWork(job.id);
+ return { source_id: src.id, method: 'remux', ...ctx.checkpoint, source_unchanged: true };
+ }
+ const source = await resolveSource(src);
+ if (!source) throw new JobError('media_unavailable', 'The source bytes are unavailable (no local file and no cloud copy)', { permanent: true });
+ checkRoom(src, Number(src.size_bytes) || 0);
+ const ext = extFor(src, source.name);
+ const out = path.join(workDir(job.id), `remux${ext}`);
+ const r = await ffmpeg(['-y', '-nostdin', '-v', 'error', ...inputArgs(source), '-i', source.input, '-map', '0:v?', '-map', '0:a?', '-c', 'copy',
+ '-fflags', '+genpts', ...(ext === '.mp4' ? ['-movflags', '+faststart'] : []), out], { signal: ctx.signal, timeoutMs: budgetMs(src.size_bytes) });
+ if (ctx.signal.aborted) throw ctx.signal.reason || new Error('aborted');
+ if (!r.ok || !existing(out) || fs.statSync(out).size === 0) {
+ cleanupWork(job.id);
+ throw new JobError('ffmpeg_failed', `ffmpeg exited ${r.code}: ${String(r.stderr).trim().split('\n').slice(-2).join(' ').slice(0, 300)}`);
+ }
+ const duration = await probeDuration(out);
+ const made = await adopt({
+ src, file: out, ext, job, variant: 'remux',
+ relation: { method: 'remux', duration_seconds: duration },
+ metadata: { duration_seconds: duration },
+ saveCheckpoint: ctx.saveCheckpoint,
+ checkpointFor: (id, size) => ({ object_id: id, size_bytes: size, duration_seconds: duration }),
+ });
+ cleanupWork(job.id);
+ return { source_id: src.id, method: 'remux', object_id: made.id, size_bytes: made.size, duration_seconds: duration, source_unchanged: true };
+}
+
+module.exports = {
+ split: { lane: 'heavy', maxAttempts: 3, timeoutMs: 6 * 3600 * 1000, needsObject: true, validate: validateSplit, run: runSplit },
+ remux: { lane: 'heavy', maxAttempts: 3, timeoutMs: 3 * 3600 * 1000, needsObject: true, validate: validateRemux, run: runRemux },
+ resolveSource, isMediaObject, extFor,
+};
new file mode 100644
new file mode 100644
@@ -0,0 +1,111 @@
@@ -0,0 +1,111 @@
+/**
+ * OpenVibe.Media — job type invariant.scan: the size-invariant validator (light lane)
+ *
+ * Scans one tenant's public playback objects (server/objects/invariant.js: ready vods/clips that are
+ * public or unlisted), records media_invariant_violations, and PROPOSES one job per object above the
+ * max (MEDIA_PUBLIC_OBJECT_MAX_MB, 500 MB):
+ *
+ * object.split when the duration is known: stream-copy parts of about the target size
+ * (MEDIA_PUBLIC_OBJECT_TARGET_MB), each a new private object derived from the source
+ * object.remux when it is not: a stream-copy remux writes the index and duration, so a split can be
+ * planned from the remuxed copy
+ *
+ * Proposals wait in status `proposed` for the owner (approve -> queued, cancel -> cancelled): nothing
+ * is split, remuxed, re-encoded or made private automatically. Idempotency key
+ * invariant:<object id>:<type>, so a later scan never proposes the same thing twice and an owner's
+ * "no" (cancelled) stands. A proposal whose object stopped violating (made private, deleted, shrunk)
+ * is withdrawn (cancelled by system:invariant.scan).
+ *
+ * result { thresholds, counts, violations, proposed, already_proposed, withdrawn, by_type }
+ */
+'use strict';
+
+const db = require('../db/database');
+const invariant = require('../objects/invariant');
+const model = require('../objects/model');
+const queue = require('./queue');
+
+const MAX_PARTS = 1000;
+const SYSTEM = 'system:invariant.scan';
+
+/** What to propose for one violating object: { type, params }. */
+function planFor(obj, t = invariant.thresholds()) {
+ const md = model.parseJson(obj.metadata, {});
+ const size = Number(obj.size_bytes) || 0;
+ const duration = Number(md.duration_seconds) || 0;
+ const target = Math.max(1, t.targetBytes);
+ const parts = Math.min(MAX_PARTS, Math.max(2, Math.ceil(size / target)));
+ const base = { source_size_bytes: size, target_bytes: t.targetBytes, max_bytes: t.maxBytes, reason: 'public_object_too_large' };
+ if (duration > 0) {
+ return { type: 'object.split', params: { ...base, parts, segment_seconds: Math.max(1, Math.ceil(duration / parts)), source_duration_seconds: duration } };
+ }
+ return { type: 'object.remux', params: { ...base, note: 'duration unknown: a stream-copy remux writes it (and the seek index), then a split can be planned' } };
+}
+
+function proposalKey(objectId, type) {
+ return `invariant:${objectId}:${type}`;
+}
+
+/**
+ * The validator. `dryRun` writes nothing (no violation rows, no proposals) and returns what it would
+ * propose. Used by the job below and by scripts/media-jobs.js scan.
+ */
+function scanTenant(appId, { dryRun = false, scanJobId = null } = {}) {
+ const report = invariant.scan({ record: !dryRun, appId });
+ const t = report.thresholds;
+ const out = {
+ thresholds: t, counts: report.counts, violations: 0, proposed: 0, already_proposed: 0, withdrawn: 0,
+ by_type: {}, dry_run: dryRun, proposals: [],
+ };
+ const violating = new Set();
+ for (const v of report.objects) {
+ if (v.level !== 'violation') continue;
+ out.violations++;
+ violating.add(v.object_id);
+ const obj = model.getObject(v.object_id);
+ if (!obj) continue;
+ const plan = planFor(obj, t);
+ out.by_type[plan.type] = (out.by_type[plan.type] || 0) + 1;
+ const key = proposalKey(obj.id, plan.type);
+ if (dryRun) {
+ const prior = db.get('SELECT id, status FROM media_jobs WHERE app_id = ? AND idempotency_key = ?', [appId, key]);
+ if (prior) out.already_proposed++; else out.proposed++;
+ out.proposals.push({ object_id: obj.id, legacy_ref: obj.legacy_ref, size_bytes: obj.size_bytes, type: plan.type, params: plan.params, existing: prior || null });
+ continue;
+ }
+ try {
+ const r = queue.enqueue({
+ appId, type: plan.type, objectId: obj.id, params: plan.params, status: 'proposed',
+ idempotencyKey: key, createdBy: scanJobId ? `${SYSTEM}:${scanJobId}` : SYSTEM,
+ });
+ if (r.created) out.proposed++; else out.already_proposed++;
+ } catch (err) {
+ if (err.code !== 'media.job.idempotency_conflict') throw err;
+ out.already_proposed++; // proposed before with other numbers: the earlier proposal (and any decision on it) stands
+ }
+ }
+ // Withdraw open proposals whose object no longer breaks the policy.
+ const open = db.all(`SELECT id, object_id FROM media_jobs WHERE app_id = ? AND status = 'proposed' AND created_by LIKE ?`, [appId, `${SYSTEM}%`]);
+ for (const p of open) {
+ if (violating.has(p.object_id)) continue;
+ if (dryRun) { out.withdrawn++; continue; }
+ const r = queue.cancel(p.id, { by: SYSTEM, reason: 'the object no longer breaks the public size policy' });
+ if (r && r.job && r.job.status === 'cancelled') out.withdrawn++;
+ }
+ if (!dryRun) delete out.proposals;
+ return out;
+}
+
+async function run(job) {
+ return scanTenant(job.app_id, { scanJobId: job.id });
+}
+
+function validate({ obj }) {
+ if (obj) throw new queue.JobError('media.job.invalid', 'invariant.scan covers the whole tenant: leave object_id out', { permanent: true });
+ return {};
+}
+
+module.exports = {
+ spec: { lane: 'light', maxAttempts: 2, timeoutMs: 10 * 60 * 1000, needsObject: false, validate, run },
+ planFor, scanTenant, proposalKey,
+};
new file mode 100644
new file mode 100644
@@ -0,0 +1,372 @@
@@ -0,0 +1,372 @@
+/**
+ * OpenVibe.Media — job queue (media_jobs; docs/object-model.md#jobs)
+ *
+ * States:
+ * proposed waiting for its owner to decide (approve -> queued, cancel -> cancelled). The worker never
+ * picks it up: the size-invariant validator proposes split/remux jobs this way.
+ * queued due when run_after has passed
+ * running held by the worker (lease_until renewed while it runs)
+ * succeeded | failed | cancelled finished
+ *
+ * Every state change and its media.job.<transition> event commit in ONE SQLite transaction (the same
+ * rule as webhooks.announce(), see events.js): an event exists if and only if its change committed.
+ * The relay is woken after the commit. Job events go to OpenVibe.Events only (no app webhook).
+ *
+ * Idempotency: a key is unique per tenant. A repeat with the same key and the same request (type,
+ * object, params) answers with the job it made; a different request under a used key is a conflict.
+ * `dedupeActive` instead joins an identical job that is still queued or running (thumbnail requests).
+ */
+'use strict';
+
+const crypto = require('crypto');
+const { EventEmitter } = require('events');
+const { ids } = require('openvibe-contracts');
+const db = require('../db/database');
+const events = require('../events');
+
+const STATUSES = ['proposed', 'queued', 'running', 'succeeded', 'failed', 'cancelled'];
+const FINISHED = ['succeeded', 'failed', 'cancelled'];
+const ACTIVE = ['queued', 'running'];
+const MAX_ACTIVE_PER_TENANT = 50; // queued + running jobs one tenant may have (proposals excluded)
+const RESULT_EVENT_MAX = 8192; // a larger result is left out of the event payload (GET the job for it)
+
+const bus = new EventEmitter();
+bus.setMaxListeners(0);
+
+class JobError extends Error {
+ /** code: stable machine code; permanent: never retried; status: HTTP status for API answers. */
+ constructor(code, message, { permanent = false, status = 400, retryAfterS = null } = {}) {
+ super(message);
+ this.code = code;
+ this.permanent = permanent;
+ this.status = status;
+ this.retryAfterS = retryAfterS;
+ }
+}
+
+// ── Types (registered by server/jobs/types.js) ───────────────
+
+const TYPES = new Map();
+let builtinsLoaded = false;
+function loadBuiltins() {
+ if (builtinsLoaded) return;
+ builtinsLoaded = true;
+ require('./types');
+}
+
+/**
+ * spec: { lane: 'light'|'heavy', maxAttempts, timeoutMs, needsObject, validate(ctx) -> params, run(job, ctx) -> result,
+ * backoffS(attempt) }
+ */
+function register(type, spec) {
+ TYPES.set(type, { lane: 'light', maxAttempts: 3, timeoutMs: 10 * 60 * 1000, needsObject: false, ...spec, type });
+}
+function typeSpec(type) {
+ loadBuiltins();
+ return TYPES.get(type) || null;
+}
+function typeNames() {
+ loadBuiltins();
+ return [...TYPES.keys()];
+}
+
+// ── Helpers ──────────────────────────────────────────────────
+
+function parseJson(s, fallback) {
+ if (s == null || s === '') return fallback;
+ try { return JSON.parse(s); } catch { return fallback; }
+}
+
+/** JSON with sorted keys, so the same params always hash the same. */
+function canonical(v) {
+ if (Array.isArray(v)) return `[${v.map(canonical).join(',')}]`;
+ if (v && typeof v === 'object') return `{${Object.keys(v).sort().map(k => `${JSON.stringify(k)}:${canonical(v[k])}`).join(',')}}`;
+ return JSON.stringify(v === undefined ? null : v);
+}
+function requestHash(type, objectId, params) {
+ return crypto.createHash('sha256').update(canonical({ type, object_id: objectId || null, params: params || {} })).digest('hex');
+}
+
+function get(id) {
+ return id ? db.get('SELECT * FROM media_jobs WHERE id = ?', [String(id)]) : null;
+}
+function getForApp(id, appId) {
+ const row = get(id);
+ return row && row.app_id === appId ? row : null;
+}
+
+function jobPublic(row) {
+ if (!row) return null;
+ return {
+ id: row.id,
+ app_id: row.app_id,
+ object_id: row.object_id || null,
+ type: row.job_type,
+ status: row.status,
+ params: parseJson(row.params, {}),
+ result: parseJson(row.result, null),
+ error: row.error || null,
+ error_code: row.error_code || null,
+ attempts: row.attempts,
+ max_attempts: row.max_attempts,
+ run_after: row.run_after || null,
+ cancel_requested: !!row.cancel_requested,
+ idempotency_key: row.idempotency_key || null,
+ created_by: row.created_by || null,
+ owner_user_id: row.owner_user_id ?? null,
+ decided_by: row.decided_by || null,
+ decided_at: row.decided_at || null,
+ created_at: row.created_at,
+ updated_at: row.updated_at,
+ started_at: row.started_at || null,
+ finished_at: row.finished_at || null,
+ };
+}
+
+function eventPayload(row) {
+ const pub = jobPublic(row);
+ if (pub.result && JSON.stringify(pub.result).length > RESULT_EVENT_MAX) pub.result = { omitted: true, reason: 'larger than the event limit; GET the job' };
+ return pub;
+}
+
+/**
+ * Run `change()` and, when it reports a change, queue media.job.<transition> for the job, in one
+ * transaction; wake the relay and tell waiters after the commit. Returns the job row, or null when
+ * `change()` changed nothing (a lost race: the caller decides what that means).
+ */
+function commit(id, transition, change) {
+ let row = null;
+ db.getDb().transaction(() => {
+ if (!change()) return;
+ row = get(id);
+ if (transition) events.recordJob(transition, eventPayload(row));
+ })();
+ if (row) {
+ if (transition) events.kick();
+ bus.emit('change', row);
+ if (FINISHED.includes(row.status)) bus.emit(`finished:${row.id}`, row);
+ }
+ return row;
+}
+
+function update(id, fromStatuses, sets, params = []) {
+ const from = Array.isArray(fromStatuses) ? fromStatuses : [fromStatuses];
+ return db.run(`UPDATE media_jobs SET ${sets}, updated_at = CURRENT_TIMESTAMP WHERE id = ? AND status IN (${from.map(() => '?').join(', ')})`,
+ [...params, id, ...from]).changes > 0;
+}
+
+// ── Enqueue ──────────────────────────────────────────────────
+
+/**
+ * enqueue({ appId, type, objectId, params, status: 'queued'|'proposed', idempotencyKey, dedupeActive,
+ * createdBy, ownerUserId, maxAttempts, runAfterS })
+ * -> { job, created, replayed, deduped }
+ * Throws JobError: media.job.unknown_type, media.job.idempotency_conflict (409), media.job.too_many (429).
+ * `params` must already be validated (routes.js / the validator do that through the type's validate()).
+ */
+function enqueue({ appId, type, objectId = null, params = {}, status = 'queued', idempotencyKey = null, dedupeActive = false,
+ createdBy = null, ownerUserId = null, maxAttempts = null, runAfterS = 0 }) {
+ const spec = typeSpec(type);
+ if (!spec) throw new JobError('media.job.unknown_type', `Unknown job type ${type}`, { status: 400, permanent: true });
+ if (!['queued', 'proposed'].includes(status)) throw new Error(`a job starts queued or proposed, not ${status}`);
+ const key = idempotencyKey == null || idempotencyKey === '' ? null : String(idempotencyKey).slice(0, 200);
+ const hash = requestHash(type, objectId, params);
+ let out = null;
+ const id = `mjob_${ids.ulid()}`;
+ db.getDb().transaction(() => {
+ if (key) {
+ const prior = db.get('SELECT * FROM media_jobs WHERE app_id = ? AND idempotency_key = ?', [appId, key]);
+ if (prior) {
+ if (prior.request_hash !== hash) {
+ throw new JobError('media.job.idempotency_conflict', 'This Idempotency-Key was used for a different job request', { status: 409, permanent: true });
+ }
+ out = { job: prior, created: false, replayed: true, deduped: false };
+ return;
+ }
+ }
+ if (dedupeActive) {
+ const same = db.get(`SELECT * FROM media_jobs WHERE app_id = ? AND job_type = ? AND request_hash = ? AND status IN ('queued', 'running')
+ ORDER BY id DESC LIMIT 1`, [appId, type, hash]);
+ if (same) { out = { job: same, created: false, replayed: false, deduped: true }; return; }
+ }
+ if (status === 'queued') {
+ const active = db.get("SELECT COUNT(*) AS n FROM media_jobs WHERE app_id = ? AND status IN ('queued', 'running')", [appId]).n;
+ if (active >= MAX_ACTIVE_PER_TENANT) {
+ throw new JobError('media.job.too_many', `At most ${MAX_ACTIVE_PER_TENANT} jobs may be queued or running per tenant`, { status: 429, retryAfterS: 60 });
+ }
+ }
+ db.run(`INSERT INTO media_jobs (id, app_id, object_id, job_type, status, idempotency_key, request_hash, params, max_attempts,
+ run_after, created_by, owner_user_id)
+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ${runAfterS > 0 ? `datetime('now', '+${Math.round(runAfterS)} seconds')` : 'NULL'}, ?, ?)`,
+ [id, appId, objectId, type, status, key, hash, JSON.stringify(params || {}),
+ Math.max(1, Math.min(10, Number(maxAttempts) || spec.maxAttempts)), createdBy, ownerUserId ?? null]);
+ const row = get(id);
+ events.recordJob(status, eventPayload(row));
+ out = { job: row, created: true, replayed: false, deduped: false };
+ })();
+ if (out.created) { events.kick(); bus.emit('change', out.job); }
+ return out;
+}
+
+// ── Owner decisions ──────────────────────────────────────────
+
+/** proposed -> queued. Returns the job, or null when it is not a proposal (any more). */
+function approve(id, { by = null } = {}) {
+ return commit(id, 'queued', () => update(id, 'proposed', "status = 'queued', decided_by = ?, decided_at = CURRENT_TIMESTAMP, run_after = NULL", [by]));
+}
+
+/**
+ * Cancel by the owner. proposed/queued -> cancelled at once. A running job gets cancel_requested and
+ * stops at its next check (the worker also aborts it in-process); { job, pending: true } until then.
+ * Finished jobs: { job, finished: true } and nothing changes.
+ */
+function cancel(id, { by = null, reason = null } = {}) {
+ const row = get(id);
+ if (!row) return null;
+ if (FINISHED.includes(row.status)) return { job: row, finished: true };
+ if (row.status === 'running') {
+ db.run('UPDATE media_jobs SET cancel_requested = 1, decided_by = COALESCE(decided_by, ?), updated_at = CURRENT_TIMESTAMP WHERE id = ?', [by, id]);
+ try { require('./worker').abort(id); } catch { /* the heartbeat sees the flag */ }
+ return { job: get(id), pending: true };
+ }
+ const done = commit(id, 'cancelled', () => update(id, ['proposed', 'queued'],
+ `status = 'cancelled', finished_at = CURRENT_TIMESTAMP, decided_by = ?, decided_at = CURRENT_TIMESTAMP,
+ error = COALESCE(?, error), error_code = CASE WHEN ? IS NULL THEN error_code ELSE 'cancelled' END`, [by, reason, reason]));
+ if (done) return { job: done };
+ return cancel(id, { by, reason }); // it moved on meanwhile (picked up): try again from its new state
+}
+
+// ── Worker side ──────────────────────────────────────────────
+
+/**
+ * Take the oldest due queued job of these types (or exactly `id`): running, attempts + 1, leased.
+ * Returns the row or null.
+ */
+function claim(types, { leaseS = 120, id = null } = {}) {
+ if (!id && !types.length) return null;
+ let row = null;
+ db.getDb().transaction(() => {
+ const next = id
+ ? db.get("SELECT id FROM media_jobs WHERE id = ? AND status = 'queued'", [id])
+ : db.get(`SELECT id FROM media_jobs WHERE status = 'queued' AND job_type IN (${types.map(() => '?').join(', ')})
+ AND (run_after IS NULL OR run_after <= datetime('now')) ORDER BY id LIMIT 1`, types);
+ if (!next) return;
+ if (!update(next.id, 'queued', `status = 'running', attempts = attempts + 1, started_at = COALESCE(started_at, CURRENT_TIMESTAMP),
+ lease_until = datetime('now', '+${Math.round(leaseS)} seconds'), run_after = NULL`)) return;
+ row = get(next.id);
+ events.recordJob('started', eventPayload(row));
+ })();
+ if (row) { events.kick(); bus.emit('change', row); }
+ return row;
+}
+
+/** Extend a running job's lease. Returns { cancelRequested } (false when it is no longer running). */
+function renew(id, { leaseS = 120 } = {}) {
+ db.run(`UPDATE media_jobs SET lease_until = datetime('now', '+${Math.round(leaseS)} seconds') WHERE id = ? AND status = 'running'`, [id]);
+ const row = get(id);
+ return { running: !!row && row.status === 'running', cancelRequested: !!(row && row.cancel_requested) };
+}
+
+/** Save handler progress; `alsoInTx()` (optional) runs in the same transaction (e.g. the object a part became). */
+function saveCheckpoint(id, checkpoint, alsoInTx = null) {
+ db.getDb().transaction(() => {
+ if (alsoInTx) alsoInTx();
+ db.run('UPDATE media_jobs SET checkpoint = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ?', [JSON.stringify(checkpoint), id]);
+ })();
+}
+
+function succeed(id, result) {
+ return commit(id, 'succeeded', () => update(id, 'running',
+ "status = 'succeeded', result = ?, error = NULL, error_code = NULL, lease_until = NULL, finished_at = CURRENT_TIMESTAMP",
+ [JSON.stringify(result ?? null)]));
+}
+
+/** A failed attempt: queued again after `retryInS` (media.job.retrying), or failed for good. */
+function fail(id, { message, code = null, retryInS = null, result = null }) {
+ const msg = String(message || 'failed').slice(0, 1000);
+ if (retryInS != null) {
+ return commit(id, 'retrying', () => update(id, 'running',
+ `status = 'queued', error = ?, error_code = ?, lease_until = NULL, run_after = datetime('now', '+${Math.max(0, Math.round(retryInS))} seconds')`,
+ [msg, code]));
+ }
+ return commit(id, 'failed', () => update(id, 'running',
+ "status = 'failed', error = ?, error_code = ?, result = COALESCE(?, result), lease_until = NULL, finished_at = CURRENT_TIMESTAMP",
+ [msg, code, result == null ? null : JSON.stringify(result)]));
+}
+
+/** A running job that stopped because its owner cancelled it. */
+function markCancelled(id, { result = null } = {}) {
+ return commit(id, 'cancelled', () => update(id, 'running',
+ "status = 'cancelled', error = 'cancelled by its owner', error_code = 'cancelled', result = COALESCE(?, result), lease_until = NULL, finished_at = CURRENT_TIMESTAMP",
+ [result == null ? null : JSON.stringify(result)]));
+}
+
+/**
+ * Running jobs nobody holds any more: every one at boot (`all`), otherwise those whose lease ran out.
+ * `except` = ids this process is running. Each is retried if it has attempts left, else failed.
+ */
+function recoverInterrupted({ all = false, except = new Set(), retryInS = 30 } = {}) {
+ const rows = db.all(`SELECT * FROM media_jobs WHERE status = 'running'${all ? '' : " AND (lease_until IS NULL OR lease_until < datetime('now'))"}`);
+ let n = 0;
+ for (const row of rows) {
+ if (except.has(row.id)) continue;
+ const why = all ? 'interrupted by a restart' : 'the worker stopped renewing its lease';
+ if (row.cancel_requested) markCancelled(row.id);
+ else if (row.attempts < row.max_attempts) fail(row.id, { message: why, code: 'interrupted', retryInS });
+ else fail(row.id, { message: why, code: 'interrupted' });
+ n++;
+ }
+ return n;
+}
+
+/** Wait until the job is finished (or `timeoutMs` passes). Resolves with the row as it is then. */
+function waitFor(id, timeoutMs = 60000) {
+ const now = get(id);
+ if (!now || FINISHED.includes(now.status)) return Promise.resolve(now);
+ return new Promise((resolve) => {
+ const ev = `finished:${id}`;
+ const done = (row) => { clearTimeout(t); bus.removeListener(ev, done); resolve(row || get(id)); };
+ const t = setTimeout(() => done(null), timeoutMs);
+ if (t.unref) t.unref();
+ bus.on(ev, done);
+ });
+}
+
+// ── Reads ────────────────────────────────────────────────────
+
+/** Cursor list, newest first. filters: status, type, objectId, ownerUserId (acting user: own jobs + jobs on own objects). */
+function list(appId, { status, type, objectId, cursor, limit = 50, actingUserId = null } = {}) {
+ const conds = ['j.app_id = ?'], params = [appId];
+ if (cursor) { conds.push('j.id < ?'); params.push(String(cursor)); }
+ if (status) { conds.push('j.status = ?'); params.push(String(status)); }
+ if (type) { conds.push('j.job_type = ?'); params.push(String(type)); }
+ if (objectId) { conds.push('j.object_id = ?'); params.push(String(objectId)); }
+ if (actingUserId != null) {
+ conds.push('(j.owner_user_id = ? OR EXISTS (SELECT 1 FROM media_objects o WHERE o.id = j.object_id AND o.owner_user_id = ?))');
+ params.push(actingUserId, actingUserId);
+ }
+ const n = Math.min(Math.max(parseInt(limit, 10) || 50, 1), 200);
+ const rows = db.all(`SELECT j.* FROM media_jobs j WHERE ${conds.join(' AND ')} ORDER BY j.id DESC LIMIT ?`, [...params, n + 1]);
+ const page = rows.slice(0, n);
+ return { jobs: page, next_cursor: rows.length > n ? page[page.length - 1].id : null, limit: n };
+}
+
+function counts() {
+ const out = Object.fromEntries(STATUSES.map(s => [s, 0]));
+ for (const r of db.all('SELECT status, COUNT(*) AS n FROM media_jobs GROUP BY status')) out[r.status] = r.n;
+ return out;
+}
+
+/** Delete finished jobs of high-volume types older than `days` (thumbnail requests). Others are kept. */
+function prune({ days = 30, types = ['thumbnail.regenerate'] } = {}) {
+ return db.run(`DELETE FROM media_jobs WHERE status IN ('succeeded', 'failed', 'cancelled') AND job_type IN (${types.map(() => '?').join(', ')})
+ AND finished_at < datetime('now', ?)`, [...types, `-${Math.max(1, days)} days`]).changes;
+}
+
+module.exports = {
+ STATUSES, FINISHED, ACTIVE, MAX_ACTIVE_PER_TENANT, JobError, bus,
+ register, typeSpec, typeNames, requestHash, parseJson,
+ get, getForApp, jobPublic, list, counts,
+ enqueue, approve, cancel,
+ claim, renew, saveCheckpoint, succeed, fail, markCancelled, recoverInterrupted, waitFor, prune,
+};
new file mode 100644
new file mode 100644
@@ -0,0 +1,137 @@
@@ -0,0 +1,137 @@
+/**
+ * OpenVibe.Media — Jobs API v2 (mounted at /api/v2/:app/jobs; docs/object-model.md#jobs)
+ *
+ * GET / cursor list, newest first (?status&type&object_id&limit&cursor)
+ * POST / { type, object_id?, params?, max_attempts? } + Idempotency-Key
+ * -> 202 { job } new | 200 { job } + Idempotent-Replayed: true
+ * GET /:jobId one job
+ * POST /:jobId/approve a proposal (status proposed) -> queued
+ * POST /:jobId/cancel cancel (also DELETE /:jobId): proposed/queued at once -> 200;
+ * running -> 202 { job } with cancel_requested until it stops; finished -> 409
+ *
+ * Auth as the objects API: the app key or a Network service token granting media.object.read (reads)
+ * or media.object.upload (create, approve, cancel) for namespace :app; a developer project's app
+ * tokens on its own tenant. With X-OV-User-Id (app key) the caller acts for one of the app's users:
+ * it sees and decides only jobs it created or jobs on objects it owns, and creates jobs on its own
+ * objects only. Errors are RFC 9457 problems.
+ */
+'use strict';
+
+const express = require('express');
+const { http } = require('openvibe-contracts');
+const model = require('../objects/model');
+const queue = require('./queue');
+const { tenantAuth, tenantCors } = require('../auth');
+
+const write = tenantAuth({ capability: 'media.object.upload' });
+const read = tenantAuth({ capability: 'media.object.read' });
+
+function problem(res, status, code, detail, extra) {
+ return http.sendProblem(res, status, code, { detail, extra });
+}
+
+function who(req) {
+ if (req.principal) return req.principal.sub;
+ return `app:${req.appId}${req.authType === 'user' ? `:user:${req.userId}` : ''}`;
+}
+
+function ownsJob(req, job) {
+ if (req.authType !== 'user') return true;
+ if (job.owner_user_id != null && job.owner_user_id === req.userId) return true;
+ const obj = job.object_id ? model.getObject(job.object_id) : null;
+ return !!(obj && obj.owner_user_id != null && obj.owner_user_id === req.userId);
+}
+
+function load(req, res) {
+ const job = queue.getForApp(String(req.params.jobId || ''), req.appId);
+ if (!job || !ownsJob(req, job)) { problem(res, 404, 'media.job.not_found', 'No such job in this namespace'); return null; }
+ return job;
+}
+
+function sendJobError(res, err) {
+ if (err instanceof queue.JobError) {
+ if (err.retryAfterS) res.set('Retry-After', String(err.retryAfterS));
+ return problem(res, err.status || 400, err.code, err.message);
+ }
+ console.error('[Jobs] API error:', err.message);
+ return problem(res, 500, 'media.job.failed_request', 'Job request failed');
+}
+
+const router = express.Router({ mergeParams: true });
+router.use(tenantCors);
+
+router.get('/', read, (req, res) => {
+ try {
+ const q = req.query;
+ if (q.cursor && !/^mjob_[0-9A-Za-z_]{1,40}$/.test(String(q.cursor))) return problem(res, 400, 'media.job.invalid', 'Bad cursor');
+ if (q.status && !queue.STATUSES.includes(String(q.status))) return problem(res, 400, 'media.job.invalid', `status must be one of ${queue.STATUSES.join(', ')}`);
+ const out = queue.list(req.appId, {
+ status: q.status, type: q.type, objectId: q.object_id, cursor: q.cursor, limit: q.limit,
+ actingUserId: req.authType === 'user' ? req.userId : null,
+ });
+ res.json({ jobs: out.jobs.map(queue.jobPublic), next_cursor: out.next_cursor, limit: out.limit });
+ } catch (err) { sendJobError(res, err); }
+});
+
+router.post('/', write, (req, res) => {
+ try {
+ const b = req.body || {};
+ const type = String(b.type || '');
+ const spec = queue.typeSpec(type);
+ if (!spec) return problem(res, 400, 'media.job.unknown_type', `type must be one of ${queue.typeNames().join(', ')}`);
+ let obj = null;
+ if (b.object_id != null && b.object_id !== '') {
+ obj = model.resolveObject(String(b.object_id), req.appId);
+ if (!obj || (req.authType === 'user' && obj.visibility === 'private' && obj.owner_user_id !== req.userId)) {
+ return problem(res, 404, 'media.object.not_found', 'No such object in this namespace');
+ }
+ if (req.authType === 'user' && obj.owner_user_id !== req.userId) return problem(res, 403, 'media.object.forbidden', 'Not your object');
+ } else if (spec.needsObject) {
+ return problem(res, 400, 'media.job.invalid', `${type} needs object_id`);
+ }
+ if (b.params != null && (typeof b.params !== 'object' || Array.isArray(b.params) || JSON.stringify(b.params).length > 16384)) {
+ return problem(res, 400, 'media.job.invalid', 'params must be an object of at most 16 KB');
+ }
+ const params = spec.validate ? spec.validate({ appId: req.appId, obj, params: b.params || {} }) : (b.params || {});
+ const key = req.headers['idempotency-key'] || b.idempotency_key || null;
+ if (key != null && !/^[\x21-\x7e]{1,200}$/.test(String(key))) return problem(res, 400, 'media.job.invalid', 'Idempotency-Key must be 1-200 visible ASCII characters');
+ const r = queue.enqueue({
+ appId: req.appId, type, objectId: obj ? obj.id : null, params, idempotencyKey: key,
+ createdBy: who(req), ownerUserId: req.authType === 'user' ? req.userId : null, maxAttempts: b.max_attempts,
+ });
+ if (r.replayed) res.set('Idempotent-Replayed', 'true');
+ else require('./worker').kick();
+ res.status(r.created ? 202 : 200).json({ job: queue.jobPublic(r.job) });
+ } catch (err) { sendJobError(res, err); }
+});
+
+router.get('/:jobId', read, (req, res) => {
+ const job = load(req, res);
+ if (job) res.json({ job: queue.jobPublic(job) });
+});
+
+router.post('/:jobId/approve', write, (req, res) => {
+ try {
+ const job = load(req, res);
+ if (!job) return;
+ if (job.status !== 'proposed') return problem(res, 409, 'media.job.not_proposed', `The job is ${job.status}; only proposals are approved`);
+ const out = queue.approve(job.id, { by: who(req) });
+ if (!out) return problem(res, 409, 'media.job.not_proposed', 'The job was decided meanwhile');
+ require('./worker').kick();
+ res.json({ job: queue.jobPublic(out) });
+ } catch (err) { sendJobError(res, err); }
+});
+
+function cancelHandler(req, res) {
+ try {
+ const job = load(req, res);
+ if (!job) return;
+ const out = queue.cancel(job.id, { by: who(req) });
+ if (out.finished) return problem(res, 409, 'media.job.finished', `The job already ${out.job.status}`);
+ res.status(out.pending ? 202 : 200).json({ job: queue.jobPublic(out.job) });
+ } catch (err) { sendJobError(res, err); }
+}
+router.post('/:jobId/cancel', write, cancelHandler);
+router.delete('/:jobId', write, cancelHandler);
+
+module.exports = router;
new file mode 100644
new file mode 100644
@@ -0,0 +1,62 @@
@@ -0,0 +1,62 @@
+/**
+ * OpenVibe.Media — job type thumbnail.regenerate (light lane)
+ *
+ * The thumbnail (re)generation POST /api/v1/:app/thumbnails/:kind/:id did inline now runs as a job:
+ * the route enqueues it (joining an identical job that is still queued or running) and waits for it,
+ * so its answer is unchanged. The v2 jobs API can queue it too, for a vod or clip object.
+ *
+ * params { kind: 'vod' | 'clip', id: <row id> } (derived from the object's legacy ref when omitted)
+ * result { url, kind, id }
+ * Failures: media_unavailable (no local file and no cloud copy; permanent), not_found (permanent),
+ * generate_failed (ffmpeg produced no frame; retried).
+ */
+'use strict';
+
+const db = require('../db/database');
+const { JobError } = require('./queue');
+
+function targetOf(obj, params = {}) {
+ let kind = params.kind != null ? String(params.kind) : null;
+ let id = params.id != null ? Number(params.id) : null;
+ if ((!kind || id == null) && obj) {
+ const ref = require('../objects/model').parseLegacyRef(obj.legacy_ref);
+ if (ref && (ref.kind === 'vod' || ref.kind === 'clip')) { kind = kind || ref.kind; id = id != null ? id : Number(ref.id); }
+ }
+ if (kind !== 'vod' && kind !== 'clip') throw new JobError('media.job.invalid', "thumbnail.regenerate needs a vod or clip (params.kind 'vod' | 'clip' and params.id, or a vod/clip object)", { permanent: true });
+ if (!Number.isInteger(id) || id < 1) throw new JobError('media.job.invalid', 'thumbnail.regenerate needs params.id (the vod or clip id)', { permanent: true });
+ return { kind, id };
+}
+
+function rowOf(appId, kind, id) {
+ return kind === 'vod' ? db.getVodById(id, appId) : db.getClipById(id, appId);
+}
+
+/** Normalized params, or JobError (400). The row must exist in this tenant (and match the object when one is named). */
+function validate({ appId, obj, params }) {
+ const t = targetOf(obj, params || {});
+ const row = rowOf(appId, t.kind, t.id);
+ if (!row) throw new JobError('media.job.not_found', `${t.kind} ${t.id} not found`, { status: 404, permanent: true });
+ if (obj && row.object_id && row.object_id !== obj.id) throw new JobError('media.job.invalid', `the object is not ${t.kind} ${t.id}`, { permanent: true });
+ return t;
+}
+
+async function run(job) {
+ const { kind, id } = targetOf(null, job.params);
+ const row = rowOf(job.app_id, kind, id);
+ if (!row) throw new JobError('not_found', `${kind} ${id} not found`, { permanent: true });
+ const thumbService = require('../thumbnails/thumbnail-service');
+ let source = null;
+ if (kind === 'clip' && row.file_path && require('fs').existsSync(row.file_path)) source = { value: row.file_path };
+ else source = await require('../vod/vod-storage').resolveMediaSource(row);
+ if (!source) throw new JobError('media_unavailable', 'Media file unavailable', { permanent: true, status: 404 });
+ const url = kind === 'vod'
+ ? await thumbService.generateVodThumbnail(id, source.value)
+ : await thumbService.generateClipThumbnail(id, source.value);
+ if (!url) throw new JobError('generate_failed', 'Failed to generate thumbnail');
+ return { url, kind, id };
+}
+
+module.exports = {
+ spec: { lane: 'light', maxAttempts: 2, timeoutMs: 2 * 60 * 1000, needsObject: false, backoffS: () => 20, validate, run },
+ targetOf,
+};
new file mode 100644
new file mode 100644
@@ -0,0 +1,18 @@
@@ -0,0 +1,18 @@
+/**
+ * OpenVibe.Media — the job types the worker runs (docs/object-model.md#jobs).
+ *
+ * thumbnail.regenerate light the thumbnail (re)generation the v1 thumbnail route used to do inline
+ * invariant.scan light the size-invariant validator: proposes object.split / object.remux
+ * object.split heavy stream-copy parts of a large vod/clip as new private objects
+ * object.remux heavy a stream-copy remux of a vod/clip as a new private object
+ */
+'use strict';
+
+const queue = require('./queue');
+
+queue.register('thumbnail.regenerate', require('./thumbnail').spec);
+queue.register('invariant.scan', require('./invariant-scan').spec);
+queue.register('object.split', require('./derive').split);
+queue.register('object.remux', require('./derive').remux);
+
+module.exports = { names: () => queue.typeNames() };
new file mode 100644
new file mode 100644
@@ -0,0 +1,202 @@
@@ -0,0 +1,202 @@
+/**
+ * OpenVibe.Media — job worker (docs/object-model.md#jobs)
+ *
+ * Runs media_jobs in-process, in two lanes so a long split never holds up a thumbnail:
+ * light thumbnail.regenerate, invariant.scan MEDIA_JOBS_LIGHT_CONCURRENCY (2)
+ * heavy object.split, object.remux MEDIA_JOBS_HEAVY_CONCURRENCY (1); waits while a
+ * recording runs unless MEDIA_JOBS_HEAVY_WHILE_RECORDING=1
+ *
+ * A running job holds a lease (MEDIA_JOBS_LEASE_S) that a heartbeat renews; the heartbeat also notices
+ * an owner's cancel request. At start every job left `running` by the previous process is retried (or
+ * failed when out of attempts). A failed attempt is retried with backoff unless the handler says the
+ * failure is permanent. Handlers get { signal, checkpoint, saveCheckpoint(cp, alsoInTx) } and resume
+ * from the checkpoint on a retry.
+ *
+ * Scheduling: the size-invariant validator (invariant.scan) is enqueued per tenant every
+ * MEDIA_INVARIANT_SCAN_HOURS (it proposes split/remux jobs; it never runs them).
+ */
+'use strict';
+
+const config = require('../config');
+const db = require('../db/database');
+const queue = require('./queue');
+
+const cfg = () => config.jobs;
+const running = new Map(); // job id -> { ac, lane }
+let timer = null;
+let started = false;
+let ticking = false;
+let lastScheduleAt = 0;
+let lastPruneAt = 0;
+let kicked = false;
+
+function lanes() {
+ const types = queue.typeNames();
+ const by = (lane) => types.filter(t => queue.typeSpec(t).lane === lane);
+ return [
+ { name: 'light', types: by('light'), max: cfg().lightConcurrency },
+ { name: 'heavy', types: by('heavy'), max: cfg().heavyConcurrency },
+ ];
+}
+
+function inLane(name) {
+ let n = 0;
+ for (const r of running.values()) if (r.lane === name) n++;
+ return n;
+}
+
+function recordingActive() {
+ try { return require('../vod/recorder').activeCount() > 0; } catch { return false; }
+}
+
+function backoffS(spec, attempt) {
+ if (typeof spec.backoffS === 'function') return spec.backoffS(attempt);
+ return Math.min(3600, 30 * 4 ** Math.max(0, attempt - 1)); // 30 s, 2 min, 8 min, 32 min, 1 h
+}
+
+/** Run one claimed job to its end. Never throws. */
+async function execute(row, laneName) {
+ const spec = queue.typeSpec(row.job_type);
+ const ac = new AbortController();
+ running.set(row.id, { ac, lane: laneName });
+ const leaseS = cfg().leaseS;
+ let cancelRequested = !!row.cancel_requested;
+ const beat = setInterval(() => {
+ try {
+ const r = queue.renew(row.id, { leaseS });
+ if (r.cancelRequested && !cancelRequested) { cancelRequested = true; ac.abort(new Error('cancelled by its owner')); }
+ } catch { /* next beat */ }
+ }, Math.max(1000, Math.floor(leaseS * 1000 / 3)));
+ if (beat.unref) beat.unref();
+ const timeoutMs = spec ? spec.timeoutMs : 1000;
+ const limit = setTimeout(() => ac.abort(new Error(`timed out after ${Math.round(timeoutMs / 1000)} s`)), timeoutMs);
+ if (limit.unref) limit.unref();
+ if (cancelRequested) ac.abort(new Error('cancelled by its owner'));
+
+ const ctx = {
+ signal: ac.signal,
+ checkpoint: queue.parseJson(row.checkpoint, null),
+ attempt: row.attempts,
+ saveCheckpoint: (cp, alsoInTx) => { ctx.checkpoint = cp; queue.saveCheckpoint(row.id, cp, alsoInTx); },
+ cancelled: () => cancelRequested,
+ };
+ try {
+ if (!spec) throw new queue.JobError('media.job.unknown_type', `No handler for ${row.job_type}`, { permanent: true });
+ const result = await spec.run(queue.jobPublic(row), ctx);
+ queue.succeed(row.id, result);
+ } catch (err) {
+ const fresh = queue.get(row.id);
+ if (cancelRequested || (fresh && fresh.cancel_requested)) {
+ queue.markCancelled(row.id, { result: ctx.checkpoint ? { checkpoint: ctx.checkpoint } : null });
+ } else {
+ const permanent = !!(err && err.permanent);
+ const attempts = fresh ? fresh.attempts : row.attempts;
+ const more = !permanent && attempts < (fresh ? fresh.max_attempts : row.max_attempts);
+ const retryIn = err && err.retryAfterS != null ? err.retryAfterS : backoffS(spec || {}, attempts);
+ queue.fail(row.id, { message: (err && err.message) || String(err), code: (err && err.code) || null, retryInS: more ? retryIn : null });
+ if (!more) console.warn(`[Jobs] ${row.job_type} ${row.id} failed: ${(err && err.message) || err}`);
+ }
+ } finally {
+ clearInterval(beat);
+ clearTimeout(limit);
+ running.delete(row.id);
+ kick();
+ }
+}
+
+/**
+ * Run this queued job now, outside the lane limits: the v1 thumbnail route does this so a request
+ * is answered as promptly as before the job system (ffmpeg per request, deduplicated per item).
+ * Resolves when it finishes; resolves null when someone else already took it.
+ */
+async function runNow(id) {
+ const row = queue.claim([], { id, leaseS: cfg().leaseS });
+ if (!row) return null;
+ await execute(row, 'direct');
+ return queue.get(id);
+}
+
+function scheduleInvariantScans(nowMs = Date.now()) {
+ const hours = cfg().invariantScanHours;
+ if (!hours || hours <= 0) return 0;
+ const periodMs = hours * 3600 * 1000;
+ const period = Math.floor(nowMs / periodMs);
+ let n = 0;
+ const tenants = db.all(`SELECT DISTINCT app_id FROM media_objects WHERE kind IN ('vod', 'clip') AND visibility != 'private' AND lifecycle_status = 'ready'`);
+ for (const { app_id: appId } of tenants) {
+ try {
+ const r = queue.enqueue({ appId, type: 'invariant.scan', params: {}, idempotencyKey: `invariant.scan:${hours}h:${period}`, createdBy: 'system:schedule' });
+ if (r.created) n++;
+ } catch (err) { console.warn(`[Jobs] could not schedule invariant.scan for ${appId}: ${err.message}`); }
+ }
+ return n;
+}
+
+async function tick() {
+ if (ticking) { kicked = true; return; }
+ ticking = true;
+ try {
+ do {
+ kicked = false;
+ queue.recoverInterrupted({ except: new Set(running.keys()) });
+ const now = Date.now();
+ if (now - lastScheduleAt > 10 * 60 * 1000) { lastScheduleAt = now; scheduleInvariantScans(now); }
+ if (now - lastPruneAt > 6 * 3600 * 1000) {
+ lastPruneAt = now;
+ try { const n = queue.prune({ days: cfg().retentionDays }); if (n) console.log(`[Jobs] pruned ${n} finished thumbnail job(s)`); } catch { /* next time */ }
+ }
+ for (const lane of lanes()) {
+ if (lane.name === 'heavy' && !cfg().heavyWhileRecording && recordingActive()) continue;
+ while (inLane(lane.name) < lane.max) {
+ const row = queue.claim(lane.types, { leaseS: cfg().leaseS });
+ if (!row) break;
+ execute(row, lane.name);
+ }
+ }
+ } while (kicked);
+ } catch (err) {
+ console.warn('[Jobs] tick:', err.message);
+ } finally {
+ ticking = false;
+ }
+}
+
+/** Look for work now (after an enqueue or approval). */
+function kick() {
+ if (!started) return;
+ setImmediate(() => { tick().catch(() => {}); });
+}
+
+/** Ask a job running in this process to stop (its owner cancelled it). */
+function abort(id) {
+ const r = running.get(id);
+ if (r) r.ac.abort(new Error('cancelled by its owner'));
+ return !!r;
+}
+
+function start() {
+ if (started || !cfg().enabled) return false;
+ started = true;
+ const n = queue.recoverInterrupted({ all: true });
+ if (n) console.log(`[Jobs] ${n} job(s) interrupted by the restart were requeued or failed`);
+ lastScheduleAt = Date.now(); // the first scheduled scan waits a few minutes after boot
+ timer = setInterval(() => { tick().catch(() => {}); }, cfg().pollMs);
+ if (timer.unref) timer.unref();
+ setTimeout(() => { tick().catch(() => {}); }, 1000).unref?.();
+ console.log(`[Jobs] worker started (light ${cfg().lightConcurrency}, heavy ${cfg().heavyConcurrency}, poll ${cfg().pollMs} ms)`);
+ return true;
+}
+
+/** Stop polling and abort running jobs; they are requeued at the next start. */
+function stop() {
+ started = false;
+ if (timer) clearInterval(timer);
+ timer = null;
+ for (const r of running.values()) r.ac.abort(new Error('the service is stopping'));
+}
+
+function status() {
+ return { enabled: cfg().enabled, started, running: [...running.entries()].map(([id, r]) => ({ id, lane: r.lane })), counts: queue.counts() };
+}
+
+module.exports = { start, stop, kick, tick, abort, runNow, execute, status, scheduleInvariantScans, isStarted: () => started, _running: running };
new file mode 100644
new file mode 100644
@@ -0,0 +1,91 @@
@@ -0,0 +1,91 @@
+/**
+ * OpenVibe.Media — content-type checks for v2 uploads (docs/object-model.md#content-types)
+ *
+ * 1. The declared type must suit the kind: vod -> video/* or audio/*, clip -> video/*, thumbnail /
+ * screenshot / avatar -> image/* (not SVG). file and asset take any well-formed type.
+ * 2. Bytes that /o/:id serves inline (image/*, video/*, audio/*, application/pdf) must look like what
+ * their type says: the first bytes are checked against the container signatures below at
+ * complete. A kind with a rule is always checked.
+ * Anything else is served as an attachment with nosniff, so its declared type is taken as given.
+ */
+'use strict';
+
+const fs = require('fs');
+
+const TYPE_RE = /^[\w.+-]+\/[\w.+-]+$/;
+const KIND_RULES = {
+ vod: { re: /^(video|audio)\//, what: 'video/* or audio/*' },
+ clip: { re: /^video\//, what: 'video/*' },
+ thumbnail: { re: /^image\/(?!svg)/, what: 'image/* (not SVG)' },
+ screenshot: { re: /^image\/(?!svg)/, what: 'image/* (not SVG)' },
+ avatar: { re: /^image\/(?!svg)/, what: 'image/* (not SVG)' },
+};
+
+function normalize(type) {
+ const t = String(type || '').split(';')[0].trim().toLowerCase();
+ return TYPE_RE.test(t) ? t : null;
+}
+
+/** null when `type` suits `kind`, else the reason. */
+function kindProblem(kind, type) {
+ const rule = KIND_RULES[kind];
+ if (!rule) return null;
+ if (!type) return `a ${kind} needs a content type (${rule.what})`;
+ return rule.re.test(type) ? null : `a ${kind} must be ${rule.what}, not ${type}`;
+}
+
+/**
+ * What the first bytes are: { family: 'image' | 'video' | 'av' | 'audio' | 'pdf', format } or null.
+ * 'av' = a container that holds video, audio or both (Matroska/WebM, MP4/MOV, Ogg).
+ */
+function sniff(head) {
+ const b = Buffer.isBuffer(head) ? head : Buffer.from(head || []);
+ const at = (i, s) => b.length >= i + s.length && b.subarray(i, i + s.length).toString('latin1') === s;
+ if (b.length >= 3 && b[0] === 0xff && b[1] === 0xd8 && b[2] === 0xff) return { family: 'image', format: 'jpeg' };
+ if (at(0, '\x89PNG\r\n\x1a\n')) return { family: 'image', format: 'png' };
+ if (at(0, 'GIF87a') || at(0, 'GIF89a')) return { family: 'image', format: 'gif' };
+ if (at(0, 'RIFF') && at(8, 'WEBP')) return { family: 'image', format: 'webp' };
+ if (at(0, 'BM')) return { family: 'image', format: 'bmp' };
+ if (at(4, 'ftyp')) {
+ const brand = b.subarray(8, 12).toString('latin1');
+ if (/^(avif|avis|heic|heix|mif1|msf1)$/.test(brand)) return { family: 'image', format: brand };
+ if (/^M4A |^M4B /.test(brand)) return { family: 'audio', format: 'm4a' };
+ return { family: 'av', format: 'mp4' };
+ }
+ if (b.length >= 4 && b[0] === 0x1a && b[1] === 0x45 && b[2] === 0xdf && b[3] === 0xa3) return { family: 'av', format: 'matroska' };
+ if (at(0, 'OggS')) return { family: 'av', format: 'ogg' };
+ if (at(0, 'RIFF') && at(8, 'AVI ')) return { family: 'video', format: 'avi' };
+ if (at(0, 'RIFF') && at(8, 'WAVE')) return { family: 'audio', format: 'wav' };
+ if (at(0, 'fLaC')) return { family: 'audio', format: 'flac' };
+ if (at(0, 'ID3') || (b.length >= 2 && b[0] === 0xff && (b[1] & 0xe0) === 0xe0)) return { family: 'audio', format: 'mp3' };
+ if (b.length >= 4 && b[0] === 0x47 && b.length >= 189 && b[188] === 0x47) return { family: 'video', format: 'mpegts' };
+ if (at(0, '%PDF-')) return { family: 'pdf', format: 'pdf' };
+ return null;
+}
+
+/** Is `type` served inline (so its bytes must match it)? */
+function servedInline(type) {
+ return /^(image\/|video\/|audio\/)/.test(type || '') || type === 'application/pdf';
+}
+
+/** null when the bytes fit the declared type (or need no check), else the reason. */
+function contentProblem(kind, type, head) {
+ if (!KIND_RULES[kind] && !servedInline(type)) return null;
+ const family = String(type || '').split('/')[0];
+ const want = type === 'application/pdf' ? 'pdf' : family;
+ const got = sniff(head);
+ if (!got) return `the bytes are not a recognised ${want} format`;
+ const ok = got.family === want || (got.family === 'av' && (want === 'video' || want === 'audio'));
+ return ok ? null : `the bytes are ${got.format} (${got.family}), not ${type}`;
+}
+
+function readHead(file, n = 512) {
+ const fd = fs.openSync(file, 'r');
+ try {
+ const buf = Buffer.alloc(n);
+ const got = fs.readSync(fd, buf, 0, n, 0);
+ return buf.subarray(0, got);
+ } finally { fs.closeSync(fd); }
+}
+
+module.exports = { normalize, kindProblem, sniff, servedInline, contentProblem, readHead, KIND_RULES };
new file mode 100644
new file mode 100644
@@ -0,0 +1,264 @@
@@ -0,0 +1,264 @@
+/**
+ * OpenVibe.Media — multipart uploads of native v2 objects (docs/object-model.md#multipart-uploads)
+ *
+ * A session belongs to one `uploading` object whose size is declared. Every part but the last is
+ * exactly part_size bytes, the last is the rest, so the server knows each part's size up front and
+ * refuses any other. Parts are stored under OBJECTS_PATH/.parts/<upload id>/<n> (written to a temp
+ * file and renamed, so a dropped connection never leaves half a part) and recorded with their
+ * sha256 in media_upload_parts. Re-sending a part replaces it. That makes a session resumable: after
+ * a drop, GET the session, then send the parts in `missing`.
+ *
+ * Complete checks that every part is there (and matches the client's per-part sha256 when given),
+ * concatenates them into OBJECTS_PATH/<app>/<id> while hashing the whole object, and then the route
+ * runs the same checks as a single-part complete. Abort (or expiry after MEDIA_MULTIPART_TTL_HOURS)
+ * deletes the parts; the object stays `uploading` so it can be sent again.
+ */
+'use strict';
+
+const fs = require('fs');
+const path = require('path');
+const crypto = require('crypto');
+const { once } = require('events');
+const { ids } = require('openvibe-contracts');
+const db = require('../db/database');
+const config = require('../config');
+
+const MB = 1024 * 1024;
+const MAX_PARTS = 10000;
+
+function partsRoot() { return path.join(config.objects.path, '.parts'); }
+function sessionDir(uploadId) { return path.join(partsRoot(), uploadId); }
+function partFile(uploadId, n) { return path.join(sessionDir(uploadId), String(n)); }
+
+function getSession(uploadId) {
+ return uploadId ? db.get('SELECT * FROM media_uploads WHERE id = ?', [String(uploadId)]) : null;
+}
+
+function activeFor(objectId) {
+ return db.get("SELECT * FROM media_uploads WHERE object_id = ? AND status IN ('active', 'completing') ORDER BY created_at DESC LIMIT 1", [objectId]);
+}
+
+function listParts(uploadId) {
+ return db.all('SELECT part_number, size_bytes, sha256, received_at FROM media_upload_parts WHERE upload_id = ? ORDER BY part_number', [uploadId]);
+}
+
+/** Exact size of part n (1-based) of a session. */
+function partSize(session, n) {
+ if (n < session.parts_expected) return session.part_size;
+ return session.total_size - session.part_size * (session.parts_expected - 1);
+}
+
+function isExpired(session) {
+ return Date.parse(String(session.expires_at).replace(' ', 'T') + 'Z') < Date.now();
+}
+
+/** Part size for a total: the requested one within bounds, grown when the part count would pass 10,000. */
+function choosePartSize(total, requested) {
+ const min = config.objects.multipartMinPartMb * MB;
+ const max = config.objects.multipartMaxPartMb * MB;
+ let size = Number(requested) > 0 ? Math.floor(Number(requested)) : config.objects.multipartDefaultPartMb * MB;
+ if (size < min) return { error: `part_size must be at least ${min} bytes` };
+ if (size > max) return { error: `part_size must be at most ${max} bytes` };
+ if (Math.ceil(total / size) > MAX_PARTS) size = Math.ceil(total / MAX_PARTS);
+ if (size > max) return { error: 'the object is too large for multipart parts of the maximum size' };
+ return { size };
+}
+
+function freeBytes() {
+ try { const st = fs.statfsSync(config.objects.path); return Number(st.bavail) * Number(st.bsize); } catch { return Infinity; }
+}
+
+/**
+ * Start a session for `obj` (uploading, size declared). An active session for the object is
+ * replaced (its parts deleted). Returns the session row, or { error, status, code }.
+ */
+function initiate(obj, { partSize: requested } = {}) {
+ const total = Number(obj.size_bytes) || 0;
+ if (!total) return { status: 400, code: 'media.object.invalid', error: 'Declare size_bytes before a multipart upload' };
+ if (total > config.objects.multipartMaxMb * MB) return { status: 413, code: 'media.object.too_large', error: `Multipart uploads are limited to ${config.objects.multipartMaxMb * MB} bytes` };
+ const chosen = choosePartSize(total, requested);
+ if (chosen.error) return { status: 400, code: 'media.object.invalid', error: chosen.error };
+ // Parts and the assembled object both sit on disk until complete.
+ const need = 2 * total + config.objects.uploadMinFreeMb * MB;
+ const free = freeBytes();
+ if (free < need) return { status: 507, code: 'media.storage.insufficient', error: `Not enough free space for this upload (${Math.floor(free / MB)} MB free)` };
+ const prior = activeFor(obj.id);
+ if (prior) abort(prior.id);
+ const id = `mup_${ids.ulid()}`;
+ const parts = Math.max(1, Math.ceil(total / chosen.size));
+ db.run(`INSERT INTO media_uploads (id, object_id, app_id, part_size, total_size, parts_expected, expires_at)
+ VALUES (?, ?, ?, ?, ?, ?, datetime('now', ?))`, [id, obj.id, obj.app_id, chosen.size, total, parts, `+${Math.max(1, config.objects.multipartTtlHours)} hours`]);
+ fs.mkdirSync(sessionDir(id), { recursive: true });
+ return getSession(id);
+}
+
+/**
+ * Stream one part to disk. Resolves { part_number, size_bytes, sha256 } or { status, code, error }.
+ * `expectSha256` (X-Content-SHA256) must match when given.
+ */
+function receivePart(req, session, n, { expectSha256 = null } = {}) {
+ const want = partSize(session, n);
+ const tmp = `${partFile(session.id, n)}.${crypto.randomBytes(4).toString('hex')}.tmp`;
+ fs.mkdirSync(sessionDir(session.id), { recursive: true });
+ return new Promise((resolve) => {
+ const hash = crypto.createHash('sha256');
+ const out = fs.createWriteStream(tmp);
+ let got = 0, done = false;
+ const drop = () => { try { fs.unlinkSync(tmp); } catch { /* not written */ } };
+ const finish = (r) => { if (done) return; done = true; resolve(r); };
+ const fail = (r) => {
+ if (done) return;
+ const end = () => { drop(); finish(r); };
+ if (out.closed) end(); else { out.once('close', end); out.destroy(); }
+ };
+ req.on('data', (c) => {
+ if (done) return;
+ got += c.length;
+ if (got > want) { req.unpipe(out); req.resume(); return fail({ status: 413, code: 'media.upload.part_too_large', error: `Part ${n} is ${want} bytes` }); }
+ hash.update(c);
+ });
+ req.on('aborted', () => fail({ status: 400, code: 'media.object.upload_interrupted', error: 'Upload did not complete' }));
+ req.on('error', () => fail({ status: 400, code: 'media.object.upload_interrupted', error: 'Upload did not complete' }));
+ out.on('error', (e) => fail({ status: 500, code: 'media.object.store_failed', error: e.message }));
+ out.on('finish', () => {
+ if (done) return;
+ const sha = hash.digest('hex');
+ if (got !== want) { drop(); return finish({ status: 400, code: 'media.upload.part_size_mismatch', error: `Part ${n} must be ${want} bytes, received ${got}` }); }
+ if (expectSha256 && String(expectSha256).toLowerCase() !== sha) { drop(); return finish({ status: 400, code: 'media.upload.part_hash_mismatch', error: `Part ${n} does not match X-Content-SHA256` }); }
+ // The session may have been completed or aborted while this part streamed in.
+ const now = getSession(session.id);
+ if (!now || now.status !== 'active') { drop(); return finish({ status: 409, code: 'media.upload.not_active', error: `The upload is ${now ? now.status : 'gone'}` }); }
+ try {
+ fs.renameSync(tmp, partFile(session.id, n));
+ db.run(`INSERT INTO media_upload_parts (upload_id, part_number, size_bytes, sha256) VALUES (?, ?, ?, ?)
+ ON CONFLICT(upload_id, part_number) DO UPDATE SET size_bytes = excluded.size_bytes, sha256 = excluded.sha256, received_at = CURRENT_TIMESTAMP`,
+ [session.id, n, got, sha]);
+ db.run('UPDATE media_uploads SET updated_at = CURRENT_TIMESTAMP WHERE id = ?', [session.id]);
+ } catch (e) { drop(); return finish({ status: 500, code: 'media.object.store_failed', error: e.message }); }
+ finish({ part_number: n, size_bytes: got, sha256: sha });
+ });
+ req.pipe(out);
+ });
+}
+
+/** Public shape of a session (never paths). */
+function sessionPublic(session, { parts = true } = {}) {
+ const out = {
+ upload_id: session.id, object_id: session.object_id, status: session.status,
+ part_size: session.part_size, total_size: session.total_size, parts_expected: session.parts_expected,
+ expires_at: new Date(Date.parse(String(session.expires_at).replace(' ', 'T') + 'Z')).toISOString(),
+ created_at: session.created_at,
+ };
+ if (parts) {
+ const list = listParts(session.id);
+ const have = new Set(list.map(p => p.part_number));
+ out.parts = list;
+ out.received_bytes = list.reduce((a, p) => a + p.size_bytes, 0);
+ const missing = [];
+ for (let n = 1; n <= session.parts_expected; n++) if (!have.has(n)) missing.push(n);
+ out.missing = missing;
+ }
+ return out;
+}
+
+/** active -> completing, unless someone else got there first. */
+function beginComplete(session) {
+ return db.run("UPDATE media_uploads SET status = 'completing', updated_at = CURRENT_TIMESTAMP WHERE id = ? AND status = 'active'", [session.id]).changes > 0;
+}
+function reopen(session) {
+ db.run("UPDATE media_uploads SET status = 'active', updated_at = CURRENT_TIMESTAMP WHERE id = ? AND status = 'completing'", [session.id]);
+}
+
+/**
+ * Check the parts and concatenate them into `dest` (via a temp file), hashing the whole object.
+ * `clientParts` [{ part_number, sha256 }] (optional) must match what was received.
+ * Resolves { bytes, sha256 } or { status, code, error, missing? }.
+ */
+async function assemble(session, dest, clientParts = null) {
+ const list = listParts(session.id);
+ const byN = new Map(list.map(p => [p.part_number, p]));
+ const missing = [];
+ for (let n = 1; n <= session.parts_expected; n++) if (!byN.has(n) || !fs.existsSync(partFile(session.id, n))) missing.push(n);
+ if (missing.length) return { status: 409, code: 'media.upload.parts_missing', error: `${missing.length} part(s) missing`, missing };
+ if (Array.isArray(clientParts)) {
+ for (const c of clientParts) {
+ const p = byN.get(Number(c && c.part_number));
+ if (!p) return { status: 400, code: 'media.upload.invalid_part', error: `No part ${c && c.part_number}` };
+ if (c.sha256 && String(c.sha256).toLowerCase() !== p.sha256) return { status: 400, code: 'media.upload.part_hash_mismatch', error: `Part ${p.part_number} differs from the one received` };
+ }
+ }
+ fs.mkdirSync(path.dirname(dest), { recursive: true });
+ const tmp = `${dest}.${crypto.randomBytes(4).toString('hex')}.assembling`;
+ const hash = crypto.createHash('sha256');
+ const out = fs.createWriteStream(tmp);
+ let bytes = 0;
+ try {
+ let failed = null;
+ out.on('error', (e) => { failed = e; });
+ for (let n = 1; n <= session.parts_expected; n++) {
+ for await (const chunk of fs.createReadStream(partFile(session.id, n), { highWaterMark: 1024 * 1024 })) {
+ if (failed) throw failed;
+ hash.update(chunk);
+ bytes += chunk.length;
+ if (!out.write(chunk)) await once(out, 'drain');
+ }
+ }
+ await new Promise((resolve, reject) => { out.end((err) => (err || failed ? reject(err || failed) : resolve())); });
+ } catch (err) {
+ out.destroy();
+ try { fs.unlinkSync(tmp); } catch { /* none */ }
+ return { status: 500, code: 'media.object.store_failed', error: err.message };
+ }
+ if (bytes !== session.total_size) {
+ try { fs.unlinkSync(tmp); } catch { /* none */ }
+ return { status: 400, code: 'media.object.size_mismatch', error: `Assembled ${bytes} bytes, declared ${session.total_size}` };
+ }
+ fs.renameSync(tmp, dest);
+ return { bytes, sha256: hash.digest('hex') };
+}
+
+function finish(session) {
+ db.run("UPDATE media_uploads SET status = 'completed', completed_at = CURRENT_TIMESTAMP, updated_at = CURRENT_TIMESTAMP WHERE id = ?", [session.id]);
+ removeParts(session.id);
+}
+
+function removeParts(uploadId) {
+ try { fs.rmSync(sessionDir(uploadId), { recursive: true, force: true }); } catch { /* best effort */ }
+ db.run('DELETE FROM media_upload_parts WHERE upload_id = ?', [uploadId]);
+}
+
+/** Delete a session's parts; the object stays uploading. */
+function abort(uploadId, status = 'aborted') {
+ db.run("UPDATE media_uploads SET status = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ? AND status IN ('active', 'completing')", [status, uploadId]);
+ removeParts(uploadId);
+}
+
+/**
+ * Expire sessions past their expiry (incomplete multipart uploads) and remove part directories that
+ * no session owns. Returns { expired, orphan_dirs }.
+ */
+function purgeExpired() {
+ let expired = 0, orphans = 0;
+ for (const s of db.all("SELECT id FROM media_uploads WHERE status IN ('active', 'completing') AND expires_at < datetime('now')")) {
+ abort(s.id, 'expired');
+ expired++;
+ }
+ let dirs = [];
+ try { dirs = fs.readdirSync(partsRoot()); } catch { /* none yet */ }
+ for (const d of dirs) {
+ const s = getSession(d);
+ if (s && ['active', 'completing'].includes(s.status)) continue;
+ try { fs.rmSync(path.join(partsRoot(), d), { recursive: true, force: true }); orphans++; } catch { /* next time */ }
+ }
+ return { expired, orphan_dirs: orphans };
+}
+
+/** Sessions that are still open (reconciliation: incomplete multipart uploads). */
+function openSessions() {
+ return db.all("SELECT u.*, (SELECT COUNT(*) FROM media_upload_parts p WHERE p.upload_id = u.id) AS parts_received FROM media_uploads u WHERE status IN ('active', 'completing') ORDER BY created_at");
+}
+
+module.exports = {
+ MAX_PARTS, getSession, activeFor, listParts, partSize, isExpired, choosePartSize,
+ initiate, receivePart, sessionPublic, beginComplete, reopen, assemble, finish, abort, purgeExpired, openSessions, partFile,
+};
@@ -17,6 +17,8 @@
@@ -17,6 +17,8 @@
* orphan_location location rows whose object does not exist
* orphan_location location rows whose object does not exist
* deleted_publicly_reachable deleted object still served (legacy row, or a thumbnail of it)
* deleted_publicly_reachable deleted object still served (legacy row, or a thumbnail of it)
* missing_projection inherited rows with no object yet (run the backfill)
* missing_projection inherited rows with no object yet (run the backfill)
+ * incomplete_multipart multipart sessions still open past their expiry (the hourly purge
+ * removes their parts; one listed here means the purge is not running)
*/
*/
'use strict';
'use strict';
@@ -27,7 +29,7 @@ const model = require('./model');
@@ -27,7 +29,7 @@ const model = require('./model');
const MB = 1024 * 1024;
const MB = 1024 * 1024;
const ISSUES = ['no_canonical_location', 'canonical_missing_replica_present', 'no_present_copy', 'missing_local_file', 'remote_missing',
const ISSUES = ['no_canonical_location', 'canonical_missing_replica_present', 'no_present_copy', 'missing_local_file', 'remote_missing',
- 'size_mismatch', 'hash_mismatch', 'orphan_location', 'deleted_publicly_reachable', 'missing_projection'];
+ 'size_mismatch', 'hash_mismatch', 'orphan_location', 'deleted_publicly_reachable', 'missing_projection', 'incomplete_multipart'];
/** Real provider: undefined when the provider is not configured here (cannot verify), null on 404. */
/** Real provider: undefined when the provider is not configured here (cannot verify), null on 404. */
async function defaultHead(provider, key) {
async function defaultHead(provider, key) {
@@ -181,6 +183,11 @@ async function reconcile({ verify = false, hash = false, head = defaultHead, has
@@ -181,6 +183,11 @@ async function reconcile({ verify = false, hash = false, head = defaultHead, has
}
}
}
}
+ for (const u of db.all(`SELECT u.*, (SELECT COUNT(*) FROM media_upload_parts p WHERE p.upload_id = u.id) AS parts_received FROM media_uploads u
+ WHERE u.status IN ('active', 'completing') AND u.expires_at < datetime('now')${appId ? ' AND u.app_id = ?' : ''}`, appId ? [appId] : [])) {
+ issue('incomplete_multipart', { upload_id: u.id, object_id: u.object_id, parts_received: u.parts_received, parts_expected: u.parts_expected, expires_at: u.expires_at });
+ }
+
report.total_issues = ISSUES.reduce((n, k) => n + report.issues[k].count, 0);
report.total_issues = ISSUES.reduce((n, k) => n + report.issues[k].count, 0);
return report;
return report;
}
}
@@ -1,9 +1,16 @@
@@ -1,9 +1,16 @@
/**
/**
* OpenVibe.Media — Object API v2 (mounted at /api/v2/:app/objects; docs/object-model.md)
* OpenVibe.Media — Object API v2 (mounted at /api/v2/:app/objects; docs/object-model.md)
*
*
- * POST / init upload → { id, object, upload: { method, url, token, expires_at, max_bytes } }
- * PUT /:id/content the bytes (single part; sha256 computed; size + quota checked)
- * POST /:id/complete verify (expected hash, invariant) → lifecycle ready
+ * POST / init upload → { id, object, upload: { method, url, token, expires_at, max_bytes, … } }
+ * (objects over MEDIA_OBJECT_MAX_MB, or { multipart: true }: a multipart session)
+ * POST /:id/upload-url a fresh presigned single-PUT URL (scoped to tenant, object and size; ?ttl)
+ * PUT /:id/content the bytes (single part; sha256 computed; size + quota + content type checked)
+ * POST /:id/complete verify (expected hash, invariant, quota, content bytes) → lifecycle ready
+ * POST /:id/multipart start a multipart session { part_size } → upload_id, token
+ * GET /:id/multipart/:uploadId session: parts received (sha256), `missing` (resume)
+ * PUT /:id/multipart/:uploadId/parts/:n one part (exact size; X-Content-SHA256 optional)
+ * POST /:id/multipart/:uploadId/complete assemble + the same checks as complete → ready
+ * DELETE /:id/multipart/:uploadId abort (parts deleted; the object stays uploading)
* GET /:id metadata (id = med_… or legacy:<app>:<kind>:<id>)
* GET /:id metadata (id = med_… or legacy:<app>:<kind>:<id>)
* GET / cursor list (?kind&visibility&status&owner&user_id&include_deleted&limit&cursor)
* GET / cursor list (?kind&visibility&status&owner&user_id&include_deleted&limit&cursor)
* DELETE /:id soft delete (bytes kept MEDIA_DELETE_RETENTION_DAYS); 409 when held
* DELETE /:id soft delete (bytes kept MEDIA_DELETE_RETENTION_DAYS); 409 when held
@@ -18,8 +25,8 @@
@@ -18,8 +25,8 @@
* (:app = prj_<ULID>) take their project's app tokens only (server/auth.js); their sandbox
* (:app = prj_<ULID>) take their project's app tokens only (server/auth.js); their sandbox
* objects are served only through signed URLs, whatever their visibility. X-OV-Subject names the
* objects are served only through signed URLs, whatever their visibility. X-OV-Subject names the
* owner (usr_…); X-OV-User-Id (app key only) acts as one of the app's users. The
* owner (usr_…); X-OV-User-Id (app key only) acts as one of the app's users. The
- * content PUT and complete also accept the upload token handed out at init, so a
- * browser can send the bytes directly.
+ * content PUT and complete also accept the upload token handed out at init (a presigned
+ * URL), and the multipart routes the session token, so a browser can send the bytes directly.
*
*
* Public bytes: GET /o/:id (publicRouter) — public/unlisted objects openly,
* Public bytes: GET /o/:id (publicRouter) — public/unlisted objects openly,
* private ones only with a valid ?exp&sig from /download.
* private ones only with a valid ?exp&sig from /download.
@@ -36,6 +43,8 @@ const config = require('../config');
@@ -36,6 +43,8 @@ const config = require('../config');
const model = require('./model');
const model = require('./model');
const invariant = require('./invariant');
const invariant = require('./invariant');
const signing = require('./signing');
const signing = require('./signing');
+const ctypes = require('./content-type');
+const multipart = require('./multipart');
const { tenantAuth, tenantCors, tenantPath } = require('../auth');
const { tenantAuth, tenantCors, tenantPath } = require('../auth');
const { announce } = require('../webhooks');
const { announce } = require('../webhooks');
@@ -73,22 +82,77 @@ function load(req, res) {
@@ -73,22 +82,77 @@ function load(req, res) {
return obj;
return obj;
}
}
-/** Upload token (from init) or the usual tenant credential with media.object.upload. */
-function contentAuth(req, res, next) {
- const token = req.query.token || req.headers['x-upload-token'];
- if (!token) return upload(req, res, next);
- const id = String(req.params.id || '');
- if (!signing.verifyUploadToken(id, token)) return problem(res, 401, 'media.upload_token.invalid', 'Upload token is invalid or expired');
- // The token names the object; its tenant must be the one the URL addresses (a developer-project
- // tenant is addressed by its project id, whichever of its production/sandbox rows holds the object).
- const obj = model.getObject(id);
+/** The token-bearing request's object and tenant; the URL must address that tenant. */
+function tokenTenant(req, res, obj) {
+ // A developer-project tenant is addressed by its project id, whichever of its production/sandbox rows holds the object.
const app = obj ? db.getApp(obj.app_id) : null;
const app = obj ? db.getApp(obj.app_id) : null;
- if (!app || tenantPath(app) !== String(req.params.app || '')) return res.status(404).json({ error: 'Unknown app' });
+ if (!app || tenantPath(app) !== String(req.params.app || '')) { res.status(404).json({ error: 'Unknown app' }); return false; }
req.appId = app.app_id;
req.appId = app.app_id;
req.appPath = tenantPath(app);
req.appPath = tenantPath(app);
req.appRow = app;
req.appRow = app;
req.authType = 'upload_token';
req.authType = 'upload_token';
- next();
+ return true;
+}
+
+/** Upload token (a presigned URL from init or /upload-url) or the usual tenant credential with media.object.upload. */
+function contentAuth(req, res, next) {
+ const token = req.query.token || req.headers['x-upload-token'];
+ if (!token) return upload(req, res, next);
+ const id = String(req.params.id || '');
+ const obj = model.getObject(id);
+ // Scoped to tenant + object + size: a token for another object, tenant or size never verifies.
+ const size = obj && Number(obj.size_bytes) > 0 ? Number(obj.size_bytes) : 0;
+ if (!obj || !signing.verifyUploadToken(id, token, { tenant: obj.app_id, size })) return problem(res, 401, 'media.upload_token.invalid', 'Upload token is invalid or expired');
+ if (tokenTenant(req, res, obj)) next();
+}
+
+/** Multipart session token (from POST /:id/multipart) or the usual tenant credential with `capability`. */
+function multipartAuth(capability) {
+ const fallback = tenantAuth({ capability });
+ return (req, res, next) => {
+ const token = req.query.token || req.headers['x-upload-token'];
+ if (!token) return fallback(req, res, next);
+ const obj = model.getObject(String(req.params.id || ''));
+ const session = multipart.getSession(String(req.params.uploadId || ''));
+ const ok = obj && session && session.object_id === obj.id && signing.verifyMultipartToken(token, {
+ tenant: obj.app_id, objectId: obj.id, uploadId: session.id, totalSize: session.total_size,
+ });
+ if (!ok) return problem(res, 401, 'media.upload_token.invalid', 'Upload token is invalid or expired');
+ if (tokenTenant(req, res, obj)) next();
+ };
+}
+
+function apiBase(req) {
+ return `${config.publicUrl}/api/v2/${encodeURIComponent(req.appPath || req.appId)}/objects`;
+}
+
+/** A presigned single-PUT URL for this object: { method, url, token, expires_at, max_bytes, content_type }. */
+function presignedPut(req, obj, ttlS) {
+ const size = Number(obj.size_bytes) > 0 ? Number(obj.size_bytes) : 0;
+ const tok = signing.uploadToken(obj.id, ttlS, { tenant: obj.app_id, size });
+ return {
+ method: 'PUT',
+ url: `${apiBase(req)}/${obj.id}/content?token=${encodeURIComponent(tok.token)}`,
+ token: tok.token,
+ expires_at: tok.expires_at,
+ max_bytes: size || config.objects.maxUploadMb * MB,
+ content_type: obj.mime_type || null,
+ };
+}
+
+/** Public shape of a multipart session plus what a client needs to send it (URLs carry the session token). */
+function multipartPublic(req, obj, session, { parts = true } = {}) {
+ const tok = signing.multipartToken({ tenant: obj.app_id, objectId: obj.id, uploadId: session.id, totalSize: session.total_size, expiresAt: Date.parse(String(session.expires_at).replace(' ', 'T') + 'Z') });
+ const u = `${apiBase(req)}/${obj.id}/multipart/${session.id}`;
+ const q = `token=${encodeURIComponent(tok.token)}`;
+ return {
+ ...multipart.sessionPublic(session, { parts }),
+ token: tok.token,
+ part_url_template: `${u}/parts/{part_number}?${q}`,
+ status_url: `${u}?${q}`,
+ complete_url: `${u}/complete?${q}`,
+ abort_url: `${u}?${q}`,
+ };
}
}
function sanitizeFilename(name) {
function sanitizeFilename(name) {
@@ -138,12 +202,23 @@ async function putContent(req, res) {
@@ -138,12 +202,23 @@ async function putContent(req, res) {
if (obj.lifecycle_status !== 'uploading') { req.resume(); return problem(res, 409, 'media.object.not_uploading', `Object is ${obj.lifecycle_status}`); }
if (obj.lifecycle_status !== 'uploading') { req.resume(); return problem(res, 409, 'media.object.not_uploading', `Object is ${obj.lifecycle_status}`); }
const declared = Number(obj.size_bytes) > 0 ? Number(obj.size_bytes) : null;
const declared = Number(obj.size_bytes) > 0 ? Number(obj.size_bytes) : null;
- const cap = declared || config.objects.maxUploadMb * MB;
+ const single = config.objects.maxUploadMb * MB;
+ if (declared && declared > single) {
+ res.set('Connection', 'close'); req.resume();
+ return problem(res, 413, 'media.object.too_large', `Objects over ${single} bytes are uploaded in parts: POST /${obj.id}/multipart`);
+ }
+ if (multipart.activeFor(obj.id)) { req.resume(); return problem(res, 409, 'media.upload.multipart_active', 'A multipart upload is open for this object: complete or abort it first'); }
+ const cap = declared || single;
const len = Number(req.headers['content-length']);
const len = Number(req.headers['content-length']);
if (Number.isFinite(len) && len > cap) {
if (Number.isFinite(len) && len > cap) {
res.set('Connection', 'close'); req.resume();
res.set('Connection', 'close'); req.resume();
return problem(res, 413, 'media.object.too_large', `At most ${cap} bytes`);
return problem(res, 413, 'media.object.too_large', `At most ${cap} bytes`);
}
}
+ // No type declared at init: the request's Content-Type becomes it, and must suit the kind.
+ const headerType = ctypes.normalize(req.headers['content-type']);
+ const effectiveType = obj.mime_type || headerType || 'application/octet-stream';
+ const typeProblem = ctypes.kindProblem(obj.kind, obj.mime_type || headerType);
+ if (typeProblem) { req.resume(); return problem(res, 415, 'media.object.unsupported_type', typeProblem); }
const tmpDir = path.join(config.objects.path, '.tmp');
const tmpDir = path.join(config.objects.path, '.tmp');
fs.mkdirSync(tmpDir, { recursive: true });
fs.mkdirSync(tmpDir, { recursive: true });
@@ -163,11 +238,9 @@ async function putContent(req, res) {
@@ -163,11 +238,9 @@ async function putContent(req, res) {
const dest = model.objectFilePath(obj);
const dest = model.objectFilePath(obj);
fs.mkdirSync(path.dirname(dest), { recursive: true });
fs.mkdirSync(path.dirname(dest), { recursive: true });
fs.renameSync(tmp, dest);
fs.renameSync(tmp, dest);
- const contentType = String(req.headers['content-type'] || '').split(';')[0].trim().toLowerCase();
db.getDb().transaction(() => {
db.getDb().transaction(() => {
model.updateObject(obj.id, {
model.updateObject(obj.id, {
- size_bytes: got.bytes, content_hash: got.sha256,
- mime_type: obj.mime_type || (/^[\w.+-]+\/[\w.+-]+$/.test(contentType) ? contentType : 'application/octet-stream'),
+ size_bytes: got.bytes, content_hash: got.sha256, mime_type: effectiveType,
canonical_provider: 'local', canonical_key: dest,
canonical_provider: 'local', canonical_key: dest,
});
});
model.upsertLocation(obj.id, { provider: 'local', key: dest, state: 'present', size_bytes: got.bytes, checksum: got.sha256, verified: true });
model.upsertLocation(obj.id, { provider: 'local', key: dest, state: 'present', size_bytes: got.bytes, checksum: got.sha256, verified: true });
@@ -198,7 +271,10 @@ router.post('/', upload, (req, res) => {
@@ -198,7 +271,10 @@ router.post('/', upload, (req, res) => {
if (!Number.isInteger(size) || size < 1) return problem(res, 400, 'media.object.invalid', 'size_bytes must be a positive integer');
if (!Number.isInteger(size) || size < 1) return problem(res, 400, 'media.object.invalid', 'size_bytes must be a positive integer');
}
}
const max = config.objects.maxUploadMb * MB;
const max = config.objects.maxUploadMb * MB;
- if (size > max) return problem(res, 413, 'media.object.too_large', `Single-part uploads are limited to ${max} bytes`);
+ const wantParts = b.multipart === true || b.multipart === 'true';
+ if (wantParts && !size) return problem(res, 400, 'media.object.invalid', 'A multipart upload needs size_bytes');
+ if (!wantParts && size > max) return problem(res, 413, 'media.object.too_large', `Single-part uploads are limited to ${max} bytes; send multipart: true for larger objects`);
+ if (wantParts && size > config.objects.multipartMaxMb * MB) return problem(res, 413, 'media.object.too_large', `Objects are limited to ${config.objects.multipartMaxMb * MB} bytes`);
if (size && invariant.wouldViolate({ kind, visibility, size_bytes: size })) {
if (size && invariant.wouldViolate({ kind, visibility, size_bytes: size })) {
return problem(res, 422, 'media.invariant.public_object_too_large', `Public playback objects are limited to ${config.objects.publicMaxMb} MB`);
return problem(res, 422, 'media.invariant.public_object_too_large', `Public playback objects are limited to ${config.objects.publicMaxMb} MB`);
}
}
@@ -211,6 +287,11 @@ router.post('/', upload, (req, res) => {
@@ -211,6 +287,11 @@ router.post('/', upload, (req, res) => {
if (!quotaCheck(req, res, size)) return;
if (!quotaCheck(req, res, size)) return;
const mime = b.mime_type && /^[\w.+-]+\/[\w.+-]+$/.test(String(b.mime_type)) ? String(b.mime_type).toLowerCase() : null;
const mime = b.mime_type && /^[\w.+-]+\/[\w.+-]+$/.test(String(b.mime_type)) ? String(b.mime_type).toLowerCase() : null;
+ if (mime) {
+ const why = ctypes.kindProblem(kind, mime);
+ if (why) return problem(res, 415, 'media.object.unsupported_type', why);
+ }
+ if (b.upload_ttl != null && !(Number(b.upload_ttl) >= 60 && Number(b.upload_ttl) <= 86400)) return problem(res, 400, 'media.object.invalid', 'upload_ttl must be 60-86400 seconds');
const userId = req.authType === 'user' ? req.userId : (Number.isInteger(Number(b.user_id)) && Number(b.user_id) > 0 ? Number(b.user_id) : null);
const userId = req.authType === 'user' ? req.userId : (Number.isInteger(Number(b.user_id)) && Number(b.user_id) > 0 ? Number(b.user_id) : null);
const metadata = { ...(b.metadata || {}) };
const metadata = { ...(b.metadata || {}) };
const filename = sanitizeFilename(b.filename);
const filename = sanitizeFilename(b.filename);
@@ -222,17 +303,24 @@ router.post('/', upload, (req, res) => {
@@ -222,17 +303,24 @@ router.post('/', upload, (req, res) => {
app_id: req.appId, namespace: req.appId, kind, owner_subject: subject, owner_app: req.appId, owner_user_id: userId,
app_id: req.appId, namespace: req.appId, kind, owner_subject: subject, owner_app: req.appId, owner_user_id: userId,
visibility, lifecycle_status: 'uploading', mime_type: mime, size_bytes: size, metadata,
visibility, lifecycle_status: 'uploading', mime_type: mime, size_bytes: size, metadata,
});
});
- const tok = signing.uploadToken(id);
+ const obj = model.getObject(id);
+ let session = null;
+ if (wantParts) {
+ session = multipart.initiate(obj, { partSize: b.part_size });
+ if (session.error) {
+ db.run('DELETE FROM media_objects WHERE id = ?', [id]); // nothing was stored for it
+ return problem(res, session.status, session.code, session.error);
+ }
+ }
+ const single = !wantParts ? presignedPut(req, obj, b.upload_ttl) : null;
res.status(201).json({
res.status(201).json({
id,
id,
- object: model.objectPublic(model.getObject(id)),
+ object: model.objectPublic(obj),
upload: {
upload: {
- method: 'PUT',
- url: `${config.publicUrl}/api/v2/${encodeURIComponent(req.appPath || req.appId)}/objects/${id}/content?token=${encodeURIComponent(tok.token)}`,
- token: tok.token,
- expires_at: tok.expires_at,
- max_bytes: size || max,
- complete_url: `${config.publicUrl}/api/v2/${encodeURIComponent(req.appPath || req.appId)}/objects/${id}/complete`,
+ ...(single || { method: 'multipart', url: null, token: null, expires_at: null, max_bytes: size, content_type: mime }),
+ complete_url: `${apiBase(req)}/${id}/complete`,
+ multipart_url: `${apiBase(req)}/${id}/multipart`,
+ ...(session ? { multipart: multipartPublic(req, obj, session, { parts: false }) } : {}),
},
},
});
});
} catch (err) {
} catch (err) {
@@ -241,6 +329,45 @@ router.post('/', upload, (req, res) => {
@@ -241,6 +329,45 @@ router.post('/', upload, (req, res) => {
}
}
});
});
+/**
+ * The checks every upload ends with (single PUT and multipart): expected sha256, the public-size
+ * invariant, the quota, and whether the bytes are what their type says; then the ready transition,
+ * its invariant row and the media.object.uploaded event commit together (the webhook follows).
+ */
+function completeUpload(req, res, obj, { expectedHash = null } = {}) {
+ const loc = model.listLocations(obj.id).find(l => l.provider === 'local' && l.state === 'present');
+ if (!loc || !fs.existsSync(loc.key)) return problem(res, 409, 'media.object.no_content', 'PUT the content first');
+ const md = model.parseJson(obj.metadata, {});
+ const expected = String(expectedHash || md.expected_sha256 || '').toLowerCase();
+ if (expected && expected !== obj.content_hash) {
+ return problem(res, 422, 'media.object.hash_mismatch', 'Content hash does not match', { expected, actual: obj.content_hash });
+ }
+ if (invariant.wouldViolate(obj)) {
+ return problem(res, 422, 'media.invariant.public_object_too_large', `Public playback objects are limited to ${config.objects.publicMaxMb} MB`);
+ }
+ if (!quotaCheck(req, res, Number(obj.size_bytes) || 0, obj.id)) return;
+ const typeWhy = ctypes.kindProblem(obj.kind, obj.mime_type) || ctypes.contentProblem(obj.kind, obj.mime_type, ctypes.readHead(loc.key));
+ if (typeWhy) return problem(res, 415, 'media.object.content_mismatch', typeWhy);
+ const { data: body } = announce(req.appId, 'media.object.uploaded', {
+ change: () => {
+ model.updateObject(obj.id, { lifecycle_status: 'ready' });
+ invariant.record(model.getObject(obj.id));
+ },
+ payload: () => model.objectPublic(model.getObject(obj.id)),
+ });
+ return res.json(body);
+}
+
+/** The object named in the URL, writable by this caller, native and still uploading (else answered). */
+function loadUploading(req, res) {
+ const obj = load(req, res);
+ if (!obj) return null;
+ if (!canWrite(req, obj)) { problem(res, 403, 'media.object.forbidden', 'Not your object'); return null; }
+ if (obj.legacy_ref) { problem(res, 409, 'media.object.legacy_managed', 'This object is written through the v1 API'); return null; }
+ if (obj.lifecycle_status !== 'uploading') { problem(res, 409, 'media.object.not_uploading', `Object is ${obj.lifecycle_status}`); return null; }
+ return obj;
+}
+
router.post('/:id/complete', contentAuth, tenantCors, (req, res) => {
router.post('/:id/complete', contentAuth, tenantCors, (req, res) => {
try {
try {
const obj = load(req, res);
const obj = load(req, res);
@@ -248,33 +375,119 @@ router.post('/:id/complete', contentAuth, tenantCors, (req, res) => {
@@ -248,33 +375,119 @@ router.post('/:id/complete', contentAuth, tenantCors, (req, res) => {
if (!canWrite(req, obj)) return problem(res, 403, 'media.object.forbidden', 'Not your object');
if (!canWrite(req, obj)) return problem(res, 403, 'media.object.forbidden', 'Not your object');
if (obj.lifecycle_status === 'ready' && !obj.legacy_ref) return res.json(model.objectPublic(obj));
if (obj.lifecycle_status === 'ready' && !obj.legacy_ref) return res.json(model.objectPublic(obj));
if (obj.legacy_ref || obj.lifecycle_status !== 'uploading') return problem(res, 409, 'media.object.not_uploading', `Object is ${obj.lifecycle_status}`);
if (obj.legacy_ref || obj.lifecycle_status !== 'uploading') return problem(res, 409, 'media.object.not_uploading', `Object is ${obj.lifecycle_status}`);
- const loc = model.listLocations(obj.id).find(l => l.provider === 'local' && l.state === 'present');
- if (!loc || !fs.existsSync(loc.key)) return problem(res, 409, 'media.object.no_content', 'PUT the content first');
- const md = model.parseJson(obj.metadata, {});
- const expected = String((req.body && req.body.content_hash) || md.expected_sha256 || '').toLowerCase();
- if (expected && expected !== obj.content_hash) {
- return problem(res, 422, 'media.object.hash_mismatch', 'Content hash does not match', { expected, actual: obj.content_hash });
- }
- if (invariant.wouldViolate(obj)) {
- return problem(res, 422, 'media.invariant.public_object_too_large', `Public playback objects are limited to ${config.objects.publicMaxMb} MB`);
- }
- if (!quotaCheck(req, res, Number(obj.size_bytes) || 0, obj.id)) return;
- // The ready transition, its invariant row and the media.object.uploaded event commit
- // together; the webhook follows the commit.
- const { data: body } = announce(req.appId, 'media.object.uploaded', {
- change: () => {
- model.updateObject(obj.id, { lifecycle_status: 'ready' });
- invariant.record(model.getObject(obj.id));
- },
- payload: () => model.objectPublic(model.getObject(obj.id)),
- });
- res.json(body);
+ completeUpload(req, res, obj, { expectedHash: req.body && req.body.content_hash });
} catch (err) {
} catch (err) {
console.error('[Objects] complete error:', err.message);
console.error('[Objects] complete error:', err.message);
problem(res, 500, 'media.object.complete_failed', 'Failed to complete upload');
problem(res, 500, 'media.object.complete_failed', 'Failed to complete upload');
}
}
});
});
+// ── Presigned single-PUT URL ─────────────────────────────────
+
+router.post('/:id/upload-url', upload, (req, res) => {
+ const obj = loadUploading(req, res);
+ if (!obj) return;
+ const ttl = req.body && req.body.ttl != null ? Number(req.body.ttl) : (req.query.ttl != null ? Number(req.query.ttl) : undefined);
+ if (ttl !== undefined && !(ttl >= 60 && ttl <= 86400)) return problem(res, 400, 'media.object.invalid', 'ttl must be 60-86400 seconds');
+ if (Number(obj.size_bytes) > config.objects.maxUploadMb * MB) return problem(res, 413, 'media.object.too_large', 'This object is uploaded in parts: POST /multipart');
+ res.json(presignedPut(req, obj, ttl));
+});
+
+// ── Multipart uploads ────────────────────────────────────────
+
+const mpRead = multipartAuth('media.object.read');
+const mpWrite = multipartAuth('media.object.upload');
+
+/** The session named in the URL, belonging to the object, still open (else answered). */
+function loadSession(req, res, obj, { open = true } = {}) {
+ const session = multipart.getSession(String(req.params.uploadId || ''));
+ if (!session || session.object_id !== obj.id) { problem(res, 404, 'media.upload.not_found', 'No such upload for this object'); return null; }
+ if (open && session.status === 'active' && multipart.isExpired(session)) multipart.abort(session.id, 'expired');
+ const now = multipart.getSession(session.id);
+ if (open && now.status !== 'active') { problem(res, 409, 'media.upload.not_active', `The upload is ${now.status}`); return null; }
+ return now;
+}
+
+router.post('/:id/multipart', upload, (req, res) => {
+ try {
+ const obj = loadUploading(req, res);
+ if (!obj) return;
+ const b = req.body || {};
+ let current = obj;
+ if (!(Number(obj.size_bytes) > 0)) {
+ // Created without a size: declare it now (the same checks init makes).
+ const size = Number(b.size_bytes);
+ if (!Number.isInteger(size) || size < 1) return problem(res, 400, 'media.object.invalid', 'size_bytes (a positive integer) is required');
+ if (size > config.objects.multipartMaxMb * MB) return problem(res, 413, 'media.object.too_large', `Objects are limited to ${config.objects.multipartMaxMb * MB} bytes`);
+ if (invariant.wouldViolate({ kind: obj.kind, visibility: obj.visibility, size_bytes: size })) {
+ return problem(res, 422, 'media.invariant.public_object_too_large', `Public playback objects are limited to ${config.objects.publicMaxMb} MB`);
+ }
+ if (!quotaCheck(req, res, size, obj.id)) return;
+ model.updateObject(obj.id, { size_bytes: size });
+ current = model.getObject(obj.id);
+ } else if (b.size_bytes != null && Number(b.size_bytes) !== Number(obj.size_bytes)) {
+ return problem(res, 400, 'media.object.size_mismatch', `The object declares ${obj.size_bytes} bytes`);
+ }
+ const session = multipart.initiate(current, { partSize: b.part_size });
+ if (session.error) return problem(res, session.status, session.code, session.error);
+ res.status(201).json(multipartPublic(req, current, session, { parts: false }));
+ } catch (err) {
+ console.error('[Objects] multipart init error:', err.message);
+ problem(res, 500, 'media.upload.init_failed', 'Failed to start the upload');
+ }
+});
+
+router.get('/:id/multipart/:uploadId', mpRead, (req, res) => {
+ const obj = load(req, res);
+ if (!obj) return;
+ const session = loadSession(req, res, obj, { open: false });
+ if (!session) return;
+ if (session.status === 'active' && multipart.isExpired(session)) multipart.abort(session.id, 'expired');
+ res.json(multipart.sessionPublic(multipart.getSession(session.id)));
+});
+
+router.post('/:id/multipart/:uploadId/complete', mpWrite, tenantCors, async (req, res) => {
+ let session = null;
+ try {
+ const obj = loadUploading(req, res);
+ if (!obj) return;
+ session = loadSession(req, res, obj);
+ if (!session) return;
+ if (!multipart.beginComplete(session)) return problem(res, 409, 'media.upload.not_active', 'The upload is being completed already');
+ const b = req.body || {};
+ const dest = model.objectFilePath(obj);
+ const got = await multipart.assemble(session, dest, Array.isArray(b.parts) ? b.parts : null);
+ if (got.error) {
+ multipart.reopen(session);
+ return problem(res, got.status, got.code, got.error, got.missing ? { missing: got.missing } : undefined);
+ }
+ db.getDb().transaction(() => {
+ model.updateObject(obj.id, {
+ size_bytes: got.bytes, content_hash: got.sha256, mime_type: obj.mime_type || 'application/octet-stream',
+ canonical_provider: 'local', canonical_key: dest,
+ });
+ model.upsertLocation(obj.id, { provider: 'local', key: dest, state: 'present', size_bytes: got.bytes, checksum: got.sha256, verified: true });
+ })();
+ multipart.finish(session);
+ completeUpload(req, res, model.getObject(obj.id), { expectedHash: b.content_hash });
+ } catch (err) {
+ if (session) multipart.reopen(session);
+ console.error('[Objects] multipart complete error:', err.message);
+ if (!res.headersSent) problem(res, 500, 'media.object.complete_failed', 'Failed to complete upload');
+ }
+});
+
+router.delete('/:id/multipart/:uploadId', mpWrite, (req, res) => {
+ const obj = load(req, res);
+ if (!obj) return;
+ if (!canWrite(req, obj)) return problem(res, 403, 'media.object.forbidden', 'Not your object');
+ const session = loadSession(req, res, obj, { open: false });
+ if (!session) return;
+ if (session.status === 'completed') return problem(res, 409, 'media.upload.not_active', 'The upload is completed');
+ multipart.abort(session.id);
+ res.json(multipart.sessionPublic(multipart.getSession(session.id), { parts: false }));
+});
+
router.get('/', read, (req, res) => {
router.get('/', read, (req, res) => {
try {
try {
const q = req.query;
const q = req.query;
@@ -419,7 +632,37 @@ publicRouter.get('/:id', async (req, res) => {
@@ -419,7 +632,37 @@ publicRouter.get('/:id', async (req, res) => {
}
}
});
});
+/** PUT /:id/multipart/:uploadId/parts/:n (raw body; mounted ahead of the JSON body parser). */
+async function putPart(req, res) {
+ const obj = load(req, res);
+ if (!obj) { req.resume(); return; }
+ if (!canWrite(req, obj)) { req.resume(); return problem(res, 403, 'media.object.forbidden', 'Not your object'); }
+ if (obj.lifecycle_status !== 'uploading') { req.resume(); return problem(res, 409, 'media.object.not_uploading', `Object is ${obj.lifecycle_status}`); }
+ const session = loadSession(req, res, obj);
+ if (!session) { req.resume(); return; }
+ const n = Number(req.params.n);
+ if (!Number.isInteger(n) || n < 1 || n > session.parts_expected) {
+ req.resume();
+ return problem(res, 400, 'media.upload.invalid_part', `part_number must be 1-${session.parts_expected}`);
+ }
+ const want = multipart.partSize(session, n);
+ const len = Number(req.headers['content-length']);
+ if (Number.isFinite(len) && len !== want) {
+ res.set('Connection', 'close'); req.resume();
+ return problem(res, len > want ? 413 : 400, len > want ? 'media.upload.part_too_large' : 'media.upload.part_size_mismatch', `Part ${n} must be ${want} bytes`);
+ }
+ const expect = req.headers['x-content-sha256'];
+ if (expect != null && !/^[a-f0-9]{64}$/i.test(String(expect))) { req.resume(); return problem(res, 400, 'media.upload.invalid_part', 'X-Content-SHA256 must be a sha256 hex digest'); }
+ const got = await multipart.receivePart(req, session, n, { expectSha256: expect || null });
+ if (got.error) {
+ if (got.status === 413) res.set('Connection', 'close');
+ return problem(res, got.status, got.code, got.error);
+ }
+ res.json(got);
+}
+
module.exports = router;
module.exports = router;
module.exports.contentHandlers = [contentAuth, tenantCors, putContent];
module.exports.contentHandlers = [contentAuth, tenantCors, putContent];
+module.exports.partHandlers = [mpWrite, tenantCors, putPart];
module.exports.publicRouter = publicRouter;
module.exports.publicRouter = publicRouter;
module.exports.INLINE = INLINE;
module.exports.INLINE = INLINE;
@@ -3,7 +3,14 @@
@@ -3,7 +3,14 @@
*
*
* HMAC-SHA256 over (purpose, object id, expiry) with MEDIA_SIGNING_SECRET.
* HMAC-SHA256 over (purpose, object id, expiry) with MEDIA_SIGNING_SECRET.
* Purposes are separate so a download signature can never be replayed as an
* Purposes are separate so a download signature can never be replayed as an
- * upload token or the other way round. Without the env secret a random
+ * upload token or the other way round.
+ *
+ * Upload tokens (presigned PUT URLs) are scoped to the tenant, the object and its size:
+ * "v2.<exp>.<size>.<mac>" over (put2, tenant, object id, size, exp); size 0 = "up to the
+ * single-part limit" (the object was created without a declared size). Tokens of the earlier
+ * form "<exp>.<mac>" (object only) are still accepted until they expire. Multipart sessions
+ * get one token for all their parts: "mp1.<exp>.<mac>" over (mpart, tenant, object id,
+ * upload id, total size, exp). Without the env secret a random
* per-process secret is used (and a warning logged): links then stop working
* per-process secret is used (and a warning logged): links then stop working
* at the next restart, which is safe but not what production wants.
* at the next restart, which is safe but not what production wants.
*/
*/
@@ -67,18 +74,70 @@ function verifyFile(key, exp, sig) {
@@ -67,18 +74,70 @@ function verifyFile(key, exp, sig) {
return safeEqual(sig, mac('getf', `file:${key}`, e));
return safeEqual(sig, mac('getf', `file:${key}`, e));
}
}
-/** Token for PUT /api/v2/:app/objects/:id/content: "<exp>.<mac>". */
-function uploadToken(objectId, ttlS = config.objects.uploadTokenTtlS) {
- const exp = nowS() + Math.max(60, Number(ttlS) || 3600);
- return { token: `${exp}.${mac('put', objectId, exp)}`, expires_at: new Date(exp * 1000).toISOString() };
+function macOf(parts) {
+ return crypto.createHmac('sha256', secret()).update(parts.map(String).join('\n')).digest('base64url');
+}
+
+/** Upload token lifetime: 60 s to 24 h (default MEDIA_UPLOAD_TOKEN_TTL_S). */
+function uploadTtl(ttlS) {
+ return Math.min(86400, Math.max(60, Number(ttlS) || config.objects.uploadTokenTtlS || 3600));
+}
+
+/**
+ * Token for PUT /api/v2/:app/objects/:id/content (a presigned single-PUT URL carries it as ?token=):
+ * "v2.<exp>.<size>.<mac>" scoped to tenant + object + size.
+ */
+function uploadToken(objectId, ttlS = config.objects.uploadTokenTtlS, { tenant = '', size = 0 } = {}) {
+ const exp = nowS() + uploadTtl(ttlS);
+ const n = Math.max(0, Math.floor(Number(size) || 0));
+ return { token: `v2.${exp}.${n}.${macOf(['put2', tenant, objectId, n, exp])}`, expires_at: new Date(exp * 1000).toISOString(), size: n };
+}
+
+/**
+ * Check an upload token's signature for this object and tenant. Returns { ok, size } where size is
+ * the byte count the token is scoped to (signed, so it cannot be altered): 0 = up to the single-part
+ * limit, null = an earlier-form token (object only).
+ */
+function checkUploadToken(objectId, token, { tenant = '' } = {}) {
+ const t = String(token || '');
+ const v2 = /^v2\.(\d+)\.(\d+)\.([A-Za-z0-9_-]+)$/.exec(t);
+ if (v2) {
+ const exp = Number(v2[1]), n = Number(v2[2]);
+ if (exp < nowS()) return { ok: false };
+ return { ok: safeEqual(v2[3], macOf(['put2', tenant, objectId, n, exp])), size: n };
+ }
+ const m = /^(\d+)\.([A-Za-z0-9_-]+)$/.exec(t);
+ if (!m) return { ok: false };
+ const exp = Number(m[1]);
+ if (exp < nowS()) return { ok: false };
+ return { ok: safeEqual(m[2], mac('put', objectId, exp)), size: null };
+}
+
+/**
+ * Valid for this object, in this tenant, for an object of `size` bytes: a token scoped to a size
+ * is valid only while the object declares exactly that size (a size-0 token: any size up to the limit).
+ */
+function verifyUploadToken(objectId, token, { tenant = '', size } = {}) {
+ const r = checkUploadToken(objectId, token, { tenant });
+ if (!r.ok) return false;
+ return size === undefined || r.size === null || r.size === 0 || r.size === Number(size);
+}
+
+/** One token for every part (and the status, complete and abort calls) of a multipart session. */
+function multipartToken({ tenant, objectId, uploadId, totalSize, expiresAt }) {
+ const exp = Math.floor(new Date(expiresAt).getTime() / 1000);
+ return { token: `mp1.${exp}.${macOf(['mpart', tenant, objectId, uploadId, totalSize, exp])}`, expires_at: new Date(exp * 1000).toISOString() };
}
}
-function verifyUploadToken(objectId, token) {
- const m = /^(\d+)\.([A-Za-z0-9_-]+)$/.exec(String(token || ''));
+function verifyMultipartToken(token, { tenant, objectId, uploadId, totalSize }) {
+ const m = /^mp1\.(\d+)\.([A-Za-z0-9_-]+)$/.exec(String(token || ''));
if (!m) return false;
if (!m) return false;
const exp = Number(m[1]);
const exp = Number(m[1]);
if (exp < nowS()) return false;
if (exp < nowS()) return false;
- return safeEqual(m[2], mac('put', objectId, exp));
+ return safeEqual(m[2], macOf(['mpart', tenant, objectId, uploadId, totalSize, exp]));
}
}
-module.exports = { signedDownloadUrl, verifyDownload, signedFileUrl, verifyFile, uploadToken, verifyUploadToken };
+module.exports = {
+ signedDownloadUrl, verifyDownload, signedFileUrl, verifyFile,
+ uploadToken, uploadTtl, checkUploadToken, verifyUploadToken, multipartToken, verifyMultipartToken,
+};
@@ -76,6 +76,14 @@ function domainMetrics(registry, { db, recorder, events }) {
@@ -76,6 +76,14 @@ function domainMetrics(registry, { db, recorder, events }) {
return Number.isFinite(t) ? Math.floor(t / 1000) : null; // no run yet: no series
return Number.isFinite(t) ? Math.floor(t / 1000) : null; // no run yet: no series
},
},
});
});
+ registry.gauge({
+ name: 'media_jobs', help: 'Media jobs by type and status (proposed = waiting for the owner; see scripts/media-jobs.js)', labelNames: ['type', 'status'],
+ collect: () => db.all('SELECT job_type, status, COUNT(*) AS n FROM media_jobs GROUP BY job_type, status').map((r) => ({ labels: { type: r.job_type, status: r.status }, value: r.n })),
+ });
+ registry.gauge({
+ name: 'media_uploads_open', help: 'Multipart upload sessions still open (active or completing)',
+ collect: () => db.get("SELECT COUNT(*) AS n FROM media_uploads WHERE status IN ('active', 'completing')").n,
+ });
registry.gauge({
registry.gauge({
name: 'media_events_outbox', help: 'Events outbox rows waiting to reach OpenVibe.Events, and rows Events rejected', labelNames: ['status'],
name: 'media_events_outbox', help: 'Events outbox rows waiting to reach OpenVibe.Events, and rows Events rejected', labelNames: ['status'],
collect: () => {
collect: () => {
@@ -4,7 +4,11 @@
@@ -4,7 +4,11 @@
* POST /:kind/:id (upload or generate) → { url }
* POST /:kind/:id (upload or generate) → { url }
* kind = 'vod' | 'clip' → multipart 'thumbnail' (or JSON { image: base64 })
* kind = 'vod' | 'clip' → multipart 'thumbnail' (or JSON { image: base64 })
* uploads a custom image; with no image, the
* uploads a custom image; with no image, the
- * thumbnail is (re)generated from the media file.
+ * thumbnail is (re)generated from the media file by a
+ * thumbnail.regenerate job (server/jobs): the route queues
+ * it (joining one already queued or running for the same
+ * item), runs it at once and answers { url } as before.
+ * ?async=1 answers 202 { job } without waiting.
* kind = 'live'|'stream' → upload only; stored under a stable per-app/id
* kind = 'live'|'stream' → upload only; stored under a stable per-app/id
* filename so the URL survives refreshes.
* filename so the URL survives refreshes.
*
*
@@ -72,16 +76,29 @@ router.post('/:kind/:id', tenantAuth({ allowUser: true }), upload.single('thumbn
@@ -72,16 +76,29 @@ router.post('/:kind/:id', tenantAuth({ allowUser: true }), upload.single('thumbn
return res.json({ url });
return res.json({ url });
}
}
- // Generate from the media file (local or presigned remote).
- const vodStorage = require('../vod/vod-storage');
- const source = await vodStorage.resolveMediaSource(row);
- if (!source) return res.status(404).json({ error: 'Media file unavailable' });
- const url = kind === 'vod'
- ? await thumbService.generateVodThumbnail(numId, source.value)
- : await thumbService.generateClipThumbnail(numId, source.value);
- if (!url) return res.status(500).json({ error: 'Failed to generate thumbnail' });
- res.json({ url });
+ // Generate from the media file (local or presigned remote) as a thumbnail.regenerate job.
+ const queue = require('../jobs/queue');
+ const worker = require('../jobs/worker');
+ const r = queue.enqueue({
+ appId: req.appId, type: 'thumbnail.regenerate', objectId: row.object_id || null, params: { kind, id: numId },
+ dedupeActive: true, maxAttempts: 1, createdBy: `app:${req.appId}${req.authType === 'user' ? `:user:${req.userId}` : ''}`,
+ ownerUserId: req.authType === 'user' ? req.userId : null,
+ });
+ if (['1', 'true'].includes(String(req.query.async || ''))) {
+ worker.kick();
+ return res.status(202).json({ job: queue.jobPublic(r.job) });
+ }
+ if (r.created || r.job.status === 'queued') worker.runNow(r.job.id).catch(() => {}); // claims it unless someone already has
+ const done = await queue.waitFor(r.job.id, 90 * 1000);
+ if (!done || done.status === 'queued' || done.status === 'running') return res.status(202).json({ job: queue.jobPublic(done || r.job) });
+ if (done.status === 'succeeded') return res.json({ url: queue.parseJson(done.result, {}).url, job_id: done.id });
+ if (done.error_code === 'media_unavailable') return res.status(404).json({ error: 'Media file unavailable', job_id: done.id });
+ res.status(500).json({ error: 'Failed to generate thumbnail', job_id: done.id });
} catch (err) {
} catch (err) {
+ if (err && err.code === 'media.job.too_many') {
+ res.set('Retry-After', String(err.retryAfterS || 60));
+ return res.status(429).json({ error: err.message });
+ }
console.error('[Thumbnails] Error:', err.message);
console.error('[Thumbnails] Error:', err.message);
res.status(500).json({ error: 'Failed to process thumbnail' });
res.status(500).json({ error: 'Failed to process thumbnail' });
}
}
@@ -164,6 +164,10 @@ function bucketFor(name) {
@@ -164,6 +164,10 @@ function bucketFor(name) {
return (PROVIDER_ENV[name] && PROVIDER_ENV[name].bucket) || null;
return (PROVIDER_ENV[name] && PROVIDER_ENV[name].bucket) || null;
}
}
+function endpointFor(name) {
+ return (PROVIDER_ENV[name] && PROVIDER_ENV[name].endpoint) || null;
+}
+
// Object model (Wave 4): holds freeze an object's placement, and every tier move
// Object model (Wave 4): holds freeze an object's placement, and every tier move
// re-projects the VOD's media_locations. Lazy — the model requires this module.
// re-projects the VOD's media_locations. Lazy — the model requires this module.
const objects = () => require('../objects/model');
const objects = () => require('../objects/model');
@@ -1265,6 +1269,7 @@ module.exports = {
@@ -1265,6 +1269,7 @@ module.exports = {
isRemote,
isRemote,
providerConfigured,
providerConfigured,
bucketFor,
bucketFor,
+ endpointFor,
headObject,
headObject,
keyForVod,
keyForVod,
localPathForVod,
localPathForVod,
new file mode 100644
new file mode 100644
@@ -0,0 +1,226 @@
@@ -0,0 +1,226 @@
+'use strict';
+// The size-invariant validator job (invariant.scan) and the jobs it proposes. The validator lists public
+// playback objects over MEDIA_PUBLIC_OBJECT_MAX_MB and PROPOSES object.split (duration known) or
+// object.remux (duration unknown). Proposals never run by themselves: the worker leaves them alone until
+// their owner approves; a rejected proposal stays rejected; a proposal whose object stopped violating is
+// withdrawn; --dry-run writes nothing. Then, with ffmpeg, the approved split and a remux make private
+// derived objects and leave the source untouched, and a retry resumes after the checkpointed parts.
+const assert = require('assert');
+const fs = require('fs');
+const os = require('os');
+const path = require('path');
+const http = require('http');
+const { spawnSync } = require('child_process');
+
+const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-jobinv-'));
+const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+process.env.DB_PATH = path.join(tmp, 'media.db');
+process.env.VOD_PATH = dir('vods');
+process.env.CLIPS_PATH = dir('clips');
+process.env.FILES_PATH = dir('files');
+process.env.THUMBNAILS_PATH = dir('thumbnails');
+process.env.PASTES_PATH = dir('pastes');
+process.env.OBJECTS_PATH = dir('objects');
+process.env.MEDIA_JOBS_POLL_MS = '50';
+process.env.MEDIA_INVARIANT_SCAN_HOURS = '0';
+process.env.MEDIA_UPLOAD_MIN_FREE_MB = '0';
+
+const MB = 1024 * 1024;
+const stub = http.createServer((req, res) => {
+ req.resume();
+ req.on('end', () => {
+ res.setHeader('Content-Type', 'application/json');
+ if (req.url === '/oauth/token') return res.end(JSON.stringify({ access_token: 'tok', token_type: 'Bearer', expires_in: 300 }));
+ res.statusCode = 404; res.end('{}');
+ });
+});
+const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
+const waitFor = async (fn, ms = 20000) => {
+ const end = Date.now() + ms;
+ while (Date.now() < end) { if (await fn()) return true; await sleep(25); }
+ return false;
+};
+const hasFfmpeg = spawnSync('ffmpeg', ['-version']).status === 0;
+
+(async () => {
+ await new Promise((r) => stub.listen(0, '127.0.0.1', r));
+ const base = `http://127.0.0.1:${stub.address().port}`;
+ const db = require('../server/db/database');
+ const conn = db.getDb();
+ db.upsertApp({ app_id: 'live', api_key: 'live-key' });
+ const events = require('../server/events');
+ events.init({ eventsUrl: base, clientSecret: 's', networkUrl: base, intervalMs: 60000 });
+ const config = require('../server/config');
+ const model = require('../server/objects/model');
+ const queue = require('../server/jobs/queue');
+ const worker = require('../server/jobs/worker');
+ const scan = require('../server/jobs/invariant-scan');
+ const outbox = () => conn.prepare('SELECT envelope FROM event_outbox ORDER BY id').all().map((r) => JSON.parse(r.envelope));
+ const jobEvents = (id) => outbox().filter((e) => e.subject.type === 'job' && e.subject.id === id).map((e) => e.event_type);
+
+ const express = require('express');
+ const app = express();
+ app.use(express.json());
+ app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));
+ const server = http.createServer(app);
+ await new Promise((r) => server.listen(0, '127.0.0.1', r));
+ const api = `http://127.0.0.1:${server.address().port}`;
+ const call = async (method, p, { body, headers = {} } = {}) => {
+ const h = { authorization: 'Bearer live-key', ...headers };
+ if (body !== undefined) h['content-type'] = 'application/json';
+ const res = await fetch(api + p, { method, headers: h, body: body === undefined ? undefined : JSON.stringify(body) });
+ return { status: res.status, body: await res.json().catch(() => null) };
+ };
+ const mk = (o) => model.createObject({ app_id: 'live', owner_user_id: 5, lifecycle_status: 'ready', mime_type: 'video/webm', ...o });
+
+ // ── A. The validator proposes, never runs ──
+ const bigVod = mk({ kind: 'vod', visibility: 'public', size_bytes: 1536 * MB, metadata: { duration_seconds: 3600 } });
+ const noDur = mk({ kind: 'vod', visibility: 'unlisted', size_bytes: 800 * MB, metadata: {} });
+ mk({ kind: 'vod', visibility: 'private', size_bytes: 2048 * MB, metadata: { duration_seconds: 10 } });
+ mk({ kind: 'clip', visibility: 'public', size_bytes: 100 * MB });
+ mk({ kind: 'clip', visibility: 'public', size_bytes: 400 * MB, metadata: { duration_seconds: 60 } });
+
+ const before = db.get('SELECT COUNT(*) AS n FROM media_jobs').n;
+ const dry = scan.scanTenant('live', { dryRun: true });
+ assert.deepStrictEqual([dry.violations, dry.proposed, dry.by_type], [2, 2, { 'object.split': 1, 'object.remux': 1 }]);
+ assert.strictEqual(db.get('SELECT COUNT(*) AS n FROM media_jobs').n, before, '--dry-run proposes nothing');
+ assert.strictEqual(db.get('SELECT COUNT(*) AS n FROM media_invariant_violations').n, 0, '--dry-run records nothing');
+
+ const s1 = queue.enqueue({ appId: 'live', type: 'invariant.scan', createdBy: 'test' });
+ const ran = await worker.runNow(s1.job.id);
+ assert.strictEqual(ran.status, 'succeeded', ran.error);
+ const r1 = queue.jobPublic(ran).result;
+ assert.deepStrictEqual([r1.violations, r1.proposed, r1.already_proposed, r1.counts.warn], [2, 2, 0, 1]);
+ const proposals = db.all("SELECT * FROM media_jobs WHERE status = 'proposed' ORDER BY id").map(queue.jobPublic);
+ const split = proposals.find((p) => p.object_id === bigVod);
+ const remux = proposals.find((p) => p.object_id === noDur);
+ assert.strictEqual(split.type, 'object.split');
+ assert.deepStrictEqual([split.params.parts, split.params.segment_seconds, split.params.max_bytes], [6, 600, 500 * MB], '1.5 GB in parts of about the 256 MB target');
+ assert.strictEqual(remux.type, 'object.remux', 'no duration: remux first');
+ assert.strictEqual(split.idempotency_key, `invariant:${bigVod}:object.split`);
+ assert.deepStrictEqual(jobEvents(split.id), ['media.job.proposed']);
+ assert.strictEqual(outbox().find((e) => e.subject.id === split.id).priority, 'important', 'a proposal waiting for its owner is important');
+
+ worker.start();
+ for (let i = 0; i < 5; i++) { worker.kick(); await sleep(60); }
+ assert.deepStrictEqual([queue.get(split.id).status, queue.get(remux.id).status, queue.get(split.id).attempts], ['proposed', 'proposed', 0], 'the worker never runs a proposal');
+
+ const again = scan.scanTenant('live');
+ assert.deepStrictEqual([again.proposed, again.already_proposed], [0, 2], 'a later scan proposes nothing twice');
+ let r = await call('POST', `/api/v2/live/jobs/${remux.id}/cancel`);
+ assert.deepStrictEqual([r.status, r.body.job.status, r.body.job.decided_by], [200, 'cancelled', 'app:live'], 'the owner rejects a proposal');
+ scan.scanTenant('live');
+ assert.strictEqual(queue.get(remux.id).status, 'cancelled', 'a rejected proposal stays rejected');
+ assert.strictEqual(db.get('SELECT COUNT(*) AS n FROM media_jobs WHERE object_id = ?', [noDur]).n, 1, 'and is not proposed again');
+ model.updateObject(bigVod, { visibility: 'private' });
+ const w = scan.scanTenant('live');
+ assert.strictEqual(w.withdrawn, 1);
+ assert.deepStrictEqual([queue.get(split.id).status, queue.get(split.id).decided_by], ['cancelled', 'system:invariant.scan'], 'a proposal whose object stopped violating is withdrawn');
+
+ config.jobs.invariantScanHours = 24;
+ model.updateObject(bigVod, { visibility: 'public' });
+ const scheduled = worker.scheduleInvariantScans(Date.now());
+ assert.strictEqual(scheduled, 1, 'one scheduled scan per tenant with public playback objects');
+ assert.strictEqual(worker.scheduleInvariantScans(Date.now()), 0, 'once per period');
+ config.jobs.invariantScanHours = 0;
+ assert.ok(await waitFor(() => db.get("SELECT status FROM media_jobs WHERE job_type = 'invariant.scan' AND created_by = 'system:schedule'").status === 'succeeded'));
+ console.log('✅ invariant.scan proposes split/remux for public objects over the limit; nothing runs until the owner decides');
+
+ // ── B. Approved and explicit split/remux jobs (ffmpeg) ──
+ r = await call('POST', '/api/v2/live/jobs', { body: { type: 'object.split', object_id: model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: 'ready', size_bytes: 1 }), params: { parts: 2 } } });
+ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.job.invalid'], 'only media objects are split');
+ r = await call('POST', '/api/v2/live/jobs', { body: { type: 'object.split', object_id: bigVod, params: { parts: 1 } } });
+ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.job.invalid'], 'parts 2-1000');
+
+ if (!hasFfmpeg) {
+ console.log('⚠️ ffmpeg not found: split/remux execution skipped');
+ } else {
+ const src = path.join(process.env.VOD_PATH, 'rec-10.mp4');
+ const mk6 = spawnSync('ffmpeg', ['-v', 'error', '-y', '-f', 'lavfi', '-i', 'testsrc=duration=6:size=160x120:rate=10', '-f', 'lavfi', '-i', 'sine=frequency=440:duration=6',
+ '-c:v', 'mpeg4', '-g', '10', '-c:a', 'aac', '-shortest', src]);
+ assert.strictEqual(mk6.status, 0, String(mk6.stderr));
+ const size = fs.statSync(src).size;
+ conn.prepare("INSERT INTO vods (id, app_id, user_id, title, file_path, file_size, is_public, duration_seconds) VALUES (10, 'live', 5, 'Six seconds', ?, ?, 1, 6)").run(src, size);
+ const vodObj = model.sync('vod', 10).id;
+ const srcBefore = model.getObject(vodObj);
+ const locsBefore = JSON.stringify(model.listLocations(vodObj));
+
+ // The validator proposes a split once the policy says the object is too large; the owner approves it.
+ config.objects.publicMaxMb = 0;
+ config.objects.publicTargetMb = 0;
+ scan.scanTenant('live');
+ const prop = queue.jobPublic(db.get("SELECT * FROM media_jobs WHERE object_id = ? AND status = 'proposed'", [vodObj]));
+ assert.deepStrictEqual([prop.type, prop.params.segment_seconds], ['object.split', 1]);
+ r = await call('POST', `/api/v2/live/jobs/${prop.id}/approve`);
+ assert.deepStrictEqual([r.status, r.body.job.status, r.body.job.decided_by], [200, 'queued', 'app:live']);
+ assert.ok(await waitFor(() => ['succeeded', 'failed'].includes(queue.get(prop.id).status)), 'the approved split runs');
+ const done = queue.jobPublic(queue.get(prop.id));
+ assert.strictEqual(done.status, 'succeeded', done.error);
+ assert.deepStrictEqual(jobEvents(prop.id), ['media.job.proposed', 'media.job.queued', 'media.job.started', 'media.job.succeeded']);
+ assert.strictEqual(done.result.parts.length, 6, '6 s in 1 s parts');
+ let total = 0;
+ for (const p of done.result.parts) {
+ const o = model.getObject(p.object_id);
+ assert.deepStrictEqual([o.visibility, o.lifecycle_status, o.kind, o.app_id, o.owner_user_id, o.legacy_ref, o.mime_type], ['private', 'ready', 'vod', 'live', 5, null, 'video/mp4']);
+ const loc = model.listLocations(o.id)[0];
+ assert.deepStrictEqual([loc.provider, loc.state, loc.size_bytes], ['local', 'present', o.size_bytes]);
+ assert.ok(fs.existsSync(loc.key) && loc.key.startsWith(path.join(process.env.OBJECTS_PATH, 'live')));
+ const rel = db.get("SELECT * FROM media_relationships WHERE from_object_id = ? AND relation = 'derived_from'", [o.id]);
+ assert.strictEqual(rel.to_object_id, vodObj);
+ assert.deepStrictEqual([JSON.parse(rel.metadata).job_id, JSON.parse(rel.metadata).part], [prop.id, p.part]);
+ total += p.duration_seconds;
+ }
+ assert.ok(Math.abs(total - 6) < 0.5, 'the parts cover the source');
+ const srcAfter = model.getObject(vodObj);
+ assert.deepStrictEqual([srcAfter.visibility, srcAfter.size_bytes, srcAfter.lifecycle_status, srcAfter.content_hash], [srcBefore.visibility, srcBefore.size_bytes, 'ready', srcBefore.content_hash], 'the source is unchanged');
+ assert.strictEqual(JSON.stringify(model.listLocations(vodObj)), locsBefore);
+ assert.strictEqual(fs.statSync(src).size, size, 'its bytes too');
+ assert.ok(!fs.existsSync(path.join(process.env.OBJECTS_PATH, '.jobs', prop.id)), 'the work directory is removed');
+ config.objects.publicMaxMb = 500;
+ config.objects.publicTargetMb = 256;
+
+ // A retry resumes after the parts its checkpoint names.
+ const resumed = queue.enqueue({ appId: 'live', type: 'object.split', objectId: vodObj, params: { parts: 3 }, createdBy: 'test' });
+ queue.saveCheckpoint(resumed.job.id, { parts: [{ part: 1, object_id: 'med_done_before', start_seconds: 0, duration_seconds: 2, size_bytes: 10 }] });
+ assert.ok(await waitFor(() => queue.get(resumed.job.id).status === 'succeeded'));
+ const rr = queue.jobPublic(queue.get(resumed.job.id)).result;
+ assert.deepStrictEqual(rr.parts.map((p) => p.part), [1, 2, 3]);
+ assert.strictEqual(rr.parts[0].object_id, 'med_done_before', 'part 1 was not cut again');
+ assert.strictEqual(db.get("SELECT COUNT(*) AS n FROM media_relationships WHERE relation = 'derived_from' AND json_extract(metadata, '$.job_id') = ?", [resumed.job.id]).n, 2);
+
+ // A remux: one private object, the source's remux variant.
+ r = await call('POST', '/api/v2/live/jobs', { body: { type: 'object.remux', object_id: `legacy:live:vod:10` }, headers: { 'idempotency-key': 'remux-10' } });
+ assert.strictEqual(r.status, 202);
+ assert.ok(await waitFor(() => queue.get(r.body.job.id).status === 'succeeded'));
+ const rm = queue.jobPublic(queue.get(r.body.job.id)).result;
+ assert.deepStrictEqual([model.getObject(rm.object_id).visibility, Math.round(rm.duration_seconds), model.getVariant(vodObj, 'remux').derived_object_id], ['private', 6, rm.object_id]);
+ console.log('✅ an approved split and a remux make private derived objects; the source is untouched; retries resume');
+ }
+
+ // ── C. The operator script: scan (dry run by default), approve, cancel; events go to the service's outbox ──
+ worker.stop();
+ const script = (...a) => spawnSync(process.execPath, [path.join(__dirname, '../scripts/media-jobs.js'), ...a], { env: process.env, encoding: 'utf8' });
+ const jobsBefore = db.get('SELECT COUNT(*) AS n FROM media_jobs').n;
+ let sc = script('scan', '--app', 'live', '--json');
+ assert.strictEqual(sc.status, 0, sc.stderr);
+ assert.strictEqual(JSON.parse(sc.stdout).live.dry_run, true);
+ assert.strictEqual(db.get('SELECT COUNT(*) AS n FROM media_jobs').n, jobsBefore, 'scan is a dry run unless --apply');
+ const manual = queue.enqueue({ appId: 'live', type: 'object.remux', objectId: bigVod, status: 'proposed', createdBy: 'test' });
+ sc = script('approve', manual.job.id, '--by', 'owner');
+ assert.strictEqual(sc.status, 0, sc.stderr);
+ assert.deepStrictEqual([queue.get(manual.job.id).status, queue.get(manual.job.id).decided_by], ['queued', 'owner']);
+ assert.ok(jobEvents(manual.job.id).includes('media.job.queued'), "the script's change queued its event in the outbox");
+ sc = script('cancel', manual.job.id, '--by', 'owner');
+ assert.strictEqual(sc.status, 0, sc.stderr);
+ assert.strictEqual(queue.get(manual.job.id).status, 'cancelled');
+ assert.strictEqual(script('approve', 'mjob_nope').status, 1);
+ sc = script('list', '--status', 'cancelled', '--json');
+ assert.ok(JSON.parse(sc.stdout).jobs.some((j) => j.id === manual.job.id));
+ console.log('✅ scripts/media-jobs.js: scan dry-run, approve and cancel with their events');
+
+ events._reset();
+ server.close();
+ stub.close();
+ console.log('\njobs-invariant: all checks passed');
+ process.exit(0);
+})().catch((err) => { console.error(err); process.exit(1); });
new file mode 100644
new file mode 100644
@@ -0,0 +1,257 @@
@@ -0,0 +1,257 @@
+'use strict';
+// Media job system (server/jobs): the schema-only media_jobs table is rebuilt; every state change
+// commits with its media.job.* outbox row (and rolls back with it); idempotency keys; retries with
+// backoff, permanent failures, checkpoints; cancellation by the owner (queued at once, running through
+// the abort signal) and not by anyone else; interrupted jobs are requeued at start; the v1 thumbnail
+// route runs as a thumbnail.regenerate job and answers as before.
+const assert = require('assert');
+const fs = require('fs');
+const os = require('os');
+const path = require('path');
+const http = require('http');
+
+const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-jobs-'));
+const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+process.env.DB_PATH = path.join(tmp, 'media.db');
+process.env.VOD_PATH = dir('vods');
+process.env.CLIPS_PATH = dir('clips');
+process.env.FILES_PATH = dir('files');
+process.env.THUMBNAILS_PATH = dir('thumbnails');
+process.env.PASTES_PATH = dir('pastes');
+process.env.OBJECTS_PATH = dir('objects');
+process.env.MEDIA_JOBS_POLL_MS = '50';
+process.env.MEDIA_JOBS_LEASE_S = '30';
+process.env.MEDIA_INVARIANT_SCAN_HOURS = '0';
+
+// ── 1. A database from before the worker: the schema-only table is rebuilt, rows kept ──
+{
+ const Database = require('better-sqlite3');
+ const old = new Database(process.env.DB_PATH);
+ old.exec(`CREATE TABLE media_jobs (id INTEGER PRIMARY KEY AUTOINCREMENT, object_id TEXT, job_type TEXT NOT NULL,
+ status TEXT NOT NULL DEFAULT 'queued' CHECK(status IN ('queued', 'running', 'done', 'failed', 'cancelled')),
+ attempts INTEGER NOT NULL DEFAULT 0, checkpoint TEXT, error TEXT, created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
+ updated_at DATETIME DEFAULT CURRENT_TIMESTAMP);
+ CREATE INDEX idx_media_jobs_status ON media_jobs(status, job_type);
+ INSERT INTO media_jobs (object_id, job_type, status, attempts) VALUES (NULL, 'legacy.thing', 'done', 1);`);
+ old.close();
+}
+
+const hooks = [];
+const stub = http.createServer((req, res) => {
+ let body = '';
+ req.on('data', (c) => { body += c; });
+ req.on('end', () => {
+ res.setHeader('Content-Type', 'application/json');
+ if (req.url === '/oauth/token') return res.end(JSON.stringify({ access_token: 'tok', token_type: 'Bearer', expires_in: 300 }));
+ if (req.url === '/hook') { hooks.push(JSON.parse(body)); return res.end('{}'); }
+ res.statusCode = 404; res.end('{}');
+ });
+});
+
+const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
+const waitFor = async (fn, ms = 5000) => {
+ const end = Date.now() + ms;
+ while (Date.now() < end) { if (await fn()) return true; await sleep(20); }
+ return false;
+};
+
+(async () => {
+ await new Promise((r) => stub.listen(0, '127.0.0.1', r));
+ const base = `http://127.0.0.1:${stub.address().port}`;
+ const db = require('../server/db/database');
+ const raw = db.getDb();
+
+ const cols = raw.prepare('PRAGMA table_info(media_jobs)').all().map((c) => c.name);
+ assert.ok(cols.includes('app_id') && cols.includes('idempotency_key') && cols.includes('lease_until'), 'media_jobs rebuilt into the worker shape');
+ const legacy = raw.prepare("SELECT * FROM media_jobs WHERE id = 'mjob_legacy_1'").get();
+ assert.deepStrictEqual([legacy.job_type, legacy.status, legacy.app_id], ['legacy.thing', 'succeeded', 'unknown'], 'the old row is kept, done -> succeeded');
+ assert.ok(raw.prepare("SELECT 1 FROM sqlite_master WHERE name = 'idx_media_jobs_idem'").get(), 'indexes on the new columns exist');
+ db.close();
+ db.getDb(); // a second open is a no-op
+ assert.ok(db.get("SELECT 1 FROM media_jobs WHERE id = 'mjob_legacy_1'"));
+ console.log('✅ the schema-only media_jobs table is rebuilt (rows kept); reopening is idempotent');
+
+ db.upsertApp({ app_id: 'live', api_key: 'live-key', webhook_url: `${base}/hook`, webhook_secret: 's' });
+ db.upsertApp({ app_id: 'games', api_key: 'games-key' });
+ const events = require('../server/events');
+ assert.ok(events.init({ eventsUrl: base, clientSecret: 's', networkUrl: base, intervalMs: 60000 }));
+ const conn = db.getDb();
+ const outbox = () => conn.prepare('SELECT envelope FROM event_outbox ORDER BY id').all().map((r) => JSON.parse(r.envelope));
+ const jobEvents = (id) => outbox().filter((e) => e.subject.type === 'job' && e.subject.id === id).map((e) => e.event_type);
+ const failOutbox = (on) => conn.exec(on
+ ? "CREATE TEMP TRIGGER outbox_boom BEFORE INSERT ON event_outbox BEGIN SELECT RAISE(ABORT, 'outbox insert failed'); END"
+ : 'DROP TRIGGER IF EXISTS temp.outbox_boom');
+
+ const queue = require('../server/jobs/queue');
+ const worker = require('../server/jobs/worker');
+
+ // Test job types.
+ let flakyCalls = 0;
+ queue.register('test.flaky', {
+ lane: 'light', maxAttempts: 3, backoffS: () => 0,
+ run: async (job, ctx) => {
+ flakyCalls++;
+ if (!ctx.checkpoint) { ctx.saveCheckpoint({ step: 1 }); throw new Error('transient'); }
+ return { resumed_from: ctx.checkpoint.step };
+ },
+ });
+ queue.register('test.permanent', { lane: 'light', run: async () => { throw new queue.JobError('bad_input', 'will never work', { permanent: true }); } });
+ queue.register('test.always', { lane: 'light', maxAttempts: 2, backoffS: () => 0, run: async () => { throw new Error('nope'); } });
+ let release = null;
+ queue.register('test.slow', {
+ lane: 'heavy',
+ run: (job, ctx) => new Promise((resolve, reject) => {
+ release = resolve;
+ ctx.signal.addEventListener('abort', () => reject(ctx.signal.reason), { once: true });
+ }),
+ });
+
+ // ── 2. A state change and its event commit together ──
+ const a = queue.enqueue({ appId: 'live', type: 'test.flaky', params: { x: 1 }, createdBy: 'test' });
+ assert.ok(a.created && /^mjob_[0-9A-HJKMNP-TV-Z]{26}$/.test(a.job.id));
+ assert.deepStrictEqual(jobEvents(a.job.id), ['media.job.queued']);
+ const env = outbox().find((e) => e.subject.id === a.job.id);
+ assert.deepStrictEqual([env.source, env.priority, env.visibility, env.payload.type, env.payload.status, env.payload.app_id], ['media', 'low', 'internal', 'test.flaky', 'queued', 'live']);
+ const q2 = queue.enqueue({ appId: 'live', type: 'test.permanent' });
+ failOutbox(true);
+ assert.throws(() => queue.enqueue({ appId: 'live', type: 'test.flaky', params: { x: 2 } }), /outbox insert failed/);
+ assert.strictEqual(db.get("SELECT COUNT(*) AS n FROM media_jobs WHERE params = '{\"x\":2}'").n, 0, 'no job without its event');
+ assert.throws(() => queue.cancel(q2.job.id, { by: 'x' }), /outbox insert failed/);
+ failOutbox(false);
+ assert.strictEqual(queue.get(q2.job.id).status, 'queued', 'a cancel whose event cannot be written changes nothing');
+ assert.ok(hooks.length === 0, 'job events go to Events only, not the app webhook');
+ console.log('✅ job state changes commit with their media.job.* outbox row, or not at all');
+
+ // ── 3. Idempotency keys ──
+ const k1 = queue.enqueue({ appId: 'live', type: 'test.permanent', params: { n: 1 }, idempotencyKey: 'key-1' });
+ const k2 = queue.enqueue({ appId: 'live', type: 'test.permanent', params: { n: 1 }, idempotencyKey: 'key-1' });
+ assert.ok(k1.created && k2.replayed && k1.job.id === k2.job.id);
+ assert.throws(() => queue.enqueue({ appId: 'live', type: 'test.permanent', params: { n: 2 }, idempotencyKey: 'key-1' }), (e) => e.code === 'media.job.idempotency_conflict' && e.status === 409);
+ assert.ok(queue.enqueue({ appId: 'games', type: 'test.permanent', params: { n: 1 }, idempotencyKey: 'key-1' }).created, 'keys are per tenant');
+ const d1 = queue.enqueue({ appId: 'live', type: 'test.permanent', params: { same: true }, dedupeActive: true });
+ const d2 = queue.enqueue({ appId: 'live', type: 'test.permanent', params: { same: true }, dedupeActive: true });
+ assert.ok(d2.deduped && d2.job.id === d1.job.id, 'dedupeActive joins an identical queued job');
+ for (const id of [q2.job.id, k1.job.id, d1.job.id]) queue.cancel(id, { by: 'test' });
+ db.run("UPDATE media_jobs SET status = 'cancelled' WHERE app_id = 'games'");
+ console.log('✅ idempotency keys: replay, per-tenant, conflict on a different request; active-job dedupe');
+
+ // ── 4. Worker: retry with backoff + checkpoint resume, permanent failure, attempts exhausted ──
+ assert.ok(worker.start());
+ const perm = queue.enqueue({ appId: 'live', type: 'test.permanent' });
+ const always = queue.enqueue({ appId: 'live', type: 'test.always' });
+ worker.kick();
+ assert.ok(await waitFor(() => queue.get(a.job.id).status === 'succeeded'), 'the flaky job succeeds on its retry');
+ assert.strictEqual(flakyCalls, 2);
+ assert.deepStrictEqual(queue.jobPublic(queue.get(a.job.id)).result, { resumed_from: 1 }, 'the retry resumed from the checkpoint');
+ assert.deepStrictEqual(jobEvents(a.job.id), ['media.job.queued', 'media.job.started', 'media.job.retrying', 'media.job.started', 'media.job.succeeded']);
+ assert.ok(await waitFor(() => queue.get(perm.job.id).status === 'failed'));
+ assert.deepStrictEqual([queue.get(perm.job.id).attempts, queue.get(perm.job.id).error_code], [1, 'bad_input'], 'a permanent failure is not retried');
+ assert.ok(await waitFor(() => queue.get(always.job.id).status === 'failed'));
+ assert.strictEqual(queue.get(always.job.id).attempts, 2, 'retried until max_attempts');
+ assert.deepStrictEqual(jobEvents(always.job.id), ['media.job.queued', 'media.job.started', 'media.job.retrying', 'media.job.started', 'media.job.failed']);
+ const failedEv = outbox().filter((e) => e.subject.id === always.job.id).pop();
+ assert.deepStrictEqual([failedEv.priority, failedEv.payload.error], ['important', 'nope']);
+ console.log('✅ worker: retries with backoff resume from the checkpoint; permanent failures stop; attempts are capped');
+
+ // ── 5. Cancellation by the owner ──
+ const slow = queue.enqueue({ appId: 'live', type: 'test.slow', createdBy: 'app:live:user:7', ownerUserId: 7 });
+ worker.kick();
+ assert.ok(await waitFor(() => queue.get(slow.job.id).status === 'running'));
+ const express = require('express');
+ const app = express();
+ app.use(express.json());
+ app.use('/api/v2/:app/jobs', require('../server/jobs/routes'));
+ app.use('/api/v1/:app/thumbnails', require('../server/thumbnails/routes'));
+ const server = http.createServer(app);
+ await new Promise((r) => server.listen(0, '127.0.0.1', r));
+ const api = `http://127.0.0.1:${server.address().port}`;
+ const call = async (method, p, { key = 'live-key', body, headers = {} } = {}) => {
+ const h = { authorization: `Bearer ${key}`, ...headers };
+ if (body !== undefined) h['content-type'] = 'application/json';
+ const res = await fetch(api + p, { method, headers: h, body: body === undefined ? undefined : JSON.stringify(body) });
+ const text = await res.text();
+ let json = null; try { json = JSON.parse(text); } catch { /* */ }
+ return { status: res.status, body: json, headers: res.headers };
+ };
+ let r = await call('POST', `/api/v2/live/jobs/${slow.job.id}/cancel`, { headers: { 'x-ov-user-id': '8' } });
+ assert.strictEqual(r.status, 404, 'another user cannot see or cancel it');
+ r = await call('POST', `/api/v2/games/jobs/${slow.job.id}/cancel`, { key: 'games-key' });
+ assert.strictEqual(r.status, 404, 'another tenant cannot either');
+ r = await call('POST', `/api/v2/live/jobs/${slow.job.id}/cancel`, { headers: { 'x-ov-user-id': '7' } });
+ assert.deepStrictEqual([r.status, r.body.job.cancel_requested], [202, true], 'a running job stops at its next check');
+ assert.ok(await waitFor(() => queue.get(slow.job.id).status === 'cancelled'), 'the abort signal stopped it');
+ assert.deepStrictEqual(jobEvents(slow.job.id), ['media.job.queued', 'media.job.started', 'media.job.cancelled']);
+ r = await call('DELETE', `/api/v2/live/jobs/${slow.job.id}`);
+ assert.deepStrictEqual([r.status, r.body.code], [409, 'media.job.finished']);
+ const waiting = queue.enqueue({ appId: 'live', type: 'test.permanent', runAfterS: 3600 });
+ r = await call('DELETE', `/api/v2/live/jobs/${waiting.job.id}`);
+ assert.deepStrictEqual([r.status, r.body.job.status], [200, 'cancelled'], 'a queued job is cancelled at once');
+ console.log('✅ cancellation: owner only; queued at once, running through the abort signal; finished = 409');
+
+ // ── 6. Interrupted jobs are requeued at start ──
+ worker.stop();
+ const orphan = queue.enqueue({ appId: 'live', type: 'test.permanent', params: { orphan: true } });
+ db.run("UPDATE media_jobs SET status = 'running', attempts = 1, lease_until = datetime('now', '+1 hour') WHERE id = ?", [orphan.job.id]);
+ assert.strictEqual(queue.recoverInterrupted({ all: true }), 1);
+ const back = queue.get(orphan.job.id);
+ assert.deepStrictEqual([back.status, back.error_code], ['queued', 'interrupted']);
+ assert.deepStrictEqual(jobEvents(orphan.job.id).slice(-1), ['media.job.retrying']);
+ queue.cancel(orphan.job.id, { by: 'test' });
+ console.log('✅ a job left running by a dead process is requeued (or failed when out of attempts)');
+
+ // ── 7. The jobs API ──
+ r = await call('POST', '/api/v2/live/jobs', { body: { type: 'nope' } });
+ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.job.unknown_type']);
+ r = await call('POST', '/api/v2/live/jobs', { body: { type: 'object.split' } });
+ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.job.invalid'], 'object.split needs an object');
+ r = await call('POST', '/api/v2/live/jobs', { body: { type: 'invariant.scan' }, headers: { 'idempotency-key': 'scan-1' } });
+ assert.strictEqual(r.status, 202);
+ const scanId = r.body.job.id;
+ r = await call('POST', '/api/v2/live/jobs', { body: { type: 'invariant.scan' }, headers: { 'idempotency-key': 'scan-1' } });
+ assert.deepStrictEqual([r.status, r.body.job.id, r.headers.get('idempotent-replayed')], [200, scanId, 'true']);
+ r = await call('POST', `/api/v2/live/jobs/${scanId}/approve`);
+ assert.deepStrictEqual([r.status, r.body.code], [409, 'media.job.not_proposed']);
+ r = await call('GET', `/api/v2/live/jobs?type=invariant.scan`);
+ assert.deepStrictEqual(r.body.jobs.map((j) => j.id), [scanId]);
+ r = await call('GET', `/api/v2/games/jobs/${scanId}`, { key: 'games-key' });
+ assert.strictEqual(r.status, 404, 'tenants are isolated');
+ r = await call('GET', `/api/v2/live/jobs?status=bogus`);
+ assert.strictEqual(r.status, 400);
+ queue.cancel(scanId, { by: 'test' });
+ console.log('✅ jobs API: create (Idempotency-Key replay), get, list, approve only proposals, tenant isolation');
+
+ // ── 8. The v1 thumbnail route runs as a thumbnail.regenerate job ──
+ const thumbs = require('../server/thumbnails/thumbnail-service');
+ let gens = 0;
+ thumbs.generateVodThumbnail = async (id) => { gens++; await sleep(150); db.run('UPDATE vods SET thumbnail_url = ? WHERE id = ?', [`/t/vod-${id}-1.jpg`, id]); return `/t/vod-${id}-1.jpg`; };
+ fs.writeFileSync(path.join(process.env.VOD_PATH, 'rec-1.webm'), 'not really a video');
+ conn.prepare("INSERT INTO vods (id, app_id, user_id, title, file_path, is_public, duration_seconds) VALUES (1, 'live', 5, 'A', ?, 1, 60)").run(path.join(process.env.VOD_PATH, 'rec-1.webm'));
+ conn.prepare("INSERT INTO vods (id, app_id, user_id, title, file_path, is_public, duration_seconds) VALUES (2, 'live', 5, 'B', ?, 1, 60)").run(path.join(process.env.VOD_PATH, 'gone.webm'));
+ require('../server/objects/model').sync('vod', 1);
+ const [t1, t2] = await Promise.all([call('POST', '/api/v1/live/thumbnails/vod/1'), call('POST', '/api/v1/live/thumbnails/vod/1')]);
+ assert.deepStrictEqual([t1.status, t1.body.url, t2.status, t2.body.url], [200, '/t/vod-1-1.jpg', 200, '/t/vod-1-1.jpg'], 'the answer keeps its { url } shape');
+ assert.strictEqual(t1.body.job_id, t2.body.job_id, 'two concurrent requests share one job');
+ assert.strictEqual(gens, 1, 'one generation for both');
+ const tj = queue.get(t1.body.job_id);
+ assert.deepStrictEqual([tj.job_type, tj.status, tj.object_id], ['thumbnail.regenerate', 'succeeded', db.get('SELECT object_id FROM vods WHERE id = 1').object_id]);
+ assert.deepStrictEqual(jobEvents(tj.id), ['media.job.queued', 'media.job.started', 'media.job.succeeded']);
+ r = await call('POST', '/api/v1/live/thumbnails/vod/2');
+ assert.deepStrictEqual([r.status, r.body.error], [404, 'Media file unavailable'], 'no media is still a 404');
+ assert.strictEqual(queue.get(r.body.job_id).error_code, 'media_unavailable');
+ r = await call('POST', '/api/v1/live/thumbnails/vod/1?async=1');
+ assert.strictEqual(r.status, 202);
+ assert.ok(await waitFor(() => false, 50) || true);
+ worker.start();
+ assert.ok(await waitFor(() => queue.get(r.body.job.id).status === 'succeeded'), '?async=1 queues it for the worker');
+ r = await call('POST', '/api/v2/live/jobs', { body: { type: 'thumbnail.regenerate', object_id: `legacy:live:vod:1` } });
+ assert.deepStrictEqual([r.status, r.body.job.params], [202, { kind: 'vod', id: 1 }], 'the v2 API takes the object and finds the row');
+ assert.ok(await waitFor(() => queue.get(r.body.job.id).status === 'succeeded'));
+ console.log('✅ the v1 thumbnail route runs a thumbnail.regenerate job and answers { url } as before (404 without media)');
+
+ worker.stop();
+ events._reset();
+ server.close();
+ stub.close();
+ console.log('\njobs: all checks passed');
+ process.exit(0);
+})().catch((err) => { console.error(err); process.exit(1); });
new file mode 100644
new file mode 100644
@@ -0,0 +1,231 @@
@@ -0,0 +1,231 @@
+'use strict';
+// Object API v2 uploads for large files: presigned single-PUT URLs scoped to tenant, object and size
+// (MEDIA_SIGNING_SECRET, expiring); multipart initiate / part / status / complete / abort, resumable
+// after a dropped connection; content-type rules per kind and byte checks for inline-served types;
+// tenant quota and size limits; expired sessions purged.
+const assert = require('assert');
+const fs = require('fs');
+const os = require('os');
+const path = require('path');
+const http = require('http');
+const crypto = require('crypto');
+
+const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-multipart-'));
+const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+process.env.DB_PATH = path.join(tmp, 'media.db');
+process.env.VOD_PATH = dir('vods');
+process.env.FILES_PATH = dir('files');
+process.env.OBJECTS_PATH = dir('objects');
+process.env.THUMBNAILS_PATH = dir('thumbnails');
+process.env.MEDIA_PUBLIC_URL = 'https://media.test';
+process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';
+process.env.MEDIA_OBJECT_MAX_MB = '1'; // single-part limit
+process.env.MEDIA_MULTIPART_MAX_MB = '4';
+process.env.MEDIA_MULTIPART_MIN_PART_MB = '0'; // small parts in tests
+process.env.MEDIA_UPLOAD_MIN_FREE_MB = '0';
+
+const db = require('../server/db/database');
+const model = require('../server/objects/model');
+const multipart = require('../server/objects/multipart');
+const objectRoutes = require('../server/objects/routes');
+const express = require('express');
+
+db.upsertApp({ app_id: 'live', api_key: 'live-key' });
+db.upsertApp({ app_id: 'games', api_key: 'games-key' });
+db.upsertApp({ app_id: 'tiny', api_key: 'tiny-key', quota_bytes: 1024 * 1024 });
+
+const app = express();
+app.put('/api/v2/:app/objects/:id/content', ...objectRoutes.contentHandlers);
+app.put('/api/v2/:app/objects/:id/multipart/:uploadId/parts/:n', ...objectRoutes.partHandlers);
+app.use(express.json());
+app.use('/api/v2/:app/objects', objectRoutes);
+app.use('/o', objectRoutes.publicRouter);
+const server = http.createServer(app);
+const sha = (b) => crypto.createHash('sha256').update(b).digest('hex');
+const mac = (parts) => crypto.createHmac('sha256', 'test-signing-secret').update(parts.join('\n')).digest('base64url');
+
+(async () => {
+ await new Promise((r) => server.listen(0, '127.0.0.1', r));
+ const base = `http://127.0.0.1:${server.address().port}`;
+ const local = (u) => base + String(u).replace('https://media.test', '');
+ const call = async (method, p, { bearer = 'live-key', body, headers = {}, raw } = {}) => {
+ const h = { ...headers };
+ if (bearer) h.authorization = `Bearer ${bearer}`;
+ let payload;
+ if (raw !== undefined) payload = raw;
+ else if (body !== undefined) { h['content-type'] = 'application/json'; payload = JSON.stringify(body); }
+ const res = await fetch(p.startsWith('http') ? p : base + p, { method, headers: h, body: payload, redirect: 'manual' });
+ const text = await res.text();
+ let json = null; try { json = JSON.parse(text); } catch { /* not json */ }
+ return { status: res.status, body: json, text, headers: res.headers };
+ };
+ const O = '/api/v2/live/objects';
+ const now = Math.floor(Date.now() / 1000);
+
+ // ── 1. Presigned single PUT: scoped to tenant, object and size; expiring ──
+ const hello = Buffer.from('hello world');
+ let r = await call('POST', O, { body: { kind: 'file', mime_type: 'text/plain', size_bytes: hello.length, upload_ttl: 120 } });
+ assert.strictEqual(r.status, 201, JSON.stringify(r.body));
+ const a = r.body;
+ assert.ok(/^v2\.\d+\.11\.[\w-]+$/.test(a.upload.token), 'the token names its size');
+ assert.ok(Math.abs(Date.parse(a.upload.expires_at) - (Date.now() + 120000)) < 5000, 'upload_ttl sets the expiry');
+ assert.deepStrictEqual([a.upload.method, a.upload.content_type, a.upload.max_bytes], ['PUT', 'text/plain', 11]);
+ r = await call('POST', O, { body: { kind: 'file', mime_type: 'text/plain', size_bytes: hello.length } });
+ const b = r.body;
+ r = await call('PUT', local(b.upload.url).replace(b.id, a.id), { bearer: null, raw: hello });
+ assert.deepStrictEqual([r.status, r.body.code], [401, 'media.upload_token.invalid'], "another object's token");
+ const [, exp, , sig] = a.upload.token.split('.');
+ r = await call('PUT', `${O}/${a.id}/content?token=v2.${exp}.12.${sig}`, { bearer: null, raw: Buffer.from('hello world!') });
+ assert.strictEqual(r.status, 401, 'a token cannot be re-scoped to another size');
+ r = await call('PUT', local(a.upload.url).replace('/api/v2/live/', '/api/v2/games/'), { bearer: null, raw: hello });
+ assert.strictEqual(r.status, 404, 'nor used on another tenant');
+ const past = now - 5;
+ r = await call('PUT', `${O}/${a.id}/content?token=v2.${past}.11.${mac(['put2', 'live', a.id, 11, past])}`, { bearer: null, raw: hello });
+ assert.strictEqual(r.status, 401, 'expired');
+ const otherTenant = now + 100;
+ r = await call('PUT', `${O}/${a.id}/content?token=v2.${otherTenant}.11.${mac(['put2', 'games', a.id, 11, otherTenant])}`, { bearer: null, raw: hello });
+ assert.strictEqual(r.status, 401, 'a token signed for another tenant');
+ r = await call('PUT', local(a.upload.url), { bearer: null, raw: hello });
+ assert.strictEqual(r.status, 200, JSON.stringify(r.body));
+ r = await call('POST', `${O}/${a.id}/complete?token=${encodeURIComponent(a.upload.token)}`, { bearer: null });
+ assert.deepStrictEqual([r.status, r.body.lifecycle_status], [200, 'ready']);
+ r = await call('POST', `${O}/${b.id}/upload-url`, { body: { ttl: 300 } });
+ assert.ok(r.status === 200 && r.body.url.includes('token=v2.'), 'a fresh presigned URL');
+ r = await call('POST', `${O}/${b.id}/upload-url`, { body: { ttl: 5 } });
+ assert.strictEqual(r.status, 400);
+ const legacyExp = now + 300;
+ r = await call('PUT', `${O}/${b.id}/content?token=${legacyExp}.${crypto.createHmac('sha256', 'test-signing-secret').update(`put\n${b.id}\n${legacyExp}`).digest('base64url')}`, { bearer: null, raw: hello });
+ assert.strictEqual(r.status, 200, 'tokens of the earlier form work until they expire');
+ console.log('✅ presigned single-PUT URLs: scoped to tenant, object and size, expiring');
+
+ // ── 2. Content types ──
+ r = await call('POST', O, { body: { kind: 'screenshot', mime_type: 'text/html' } });
+ assert.deepStrictEqual([r.status, r.body.code], [415, 'media.object.unsupported_type']);
+ r = await call('POST', O, { body: { kind: 'vod', visibility: 'private', mime_type: 'image/png' } });
+ assert.strictEqual(r.status, 415, 'a vod is video or audio');
+ r = await call('POST', O, { body: { kind: 'avatar', mime_type: 'image/svg+xml' } });
+ assert.strictEqual(r.status, 415, 'no SVG where images are served inline');
+ r = await call('POST', O, { body: { kind: 'screenshot', mime_type: 'image/png' } });
+ const fake = r.body.id;
+ r = await call('PUT', `${O}/${fake}/content`, { raw: Buffer.from('<html><script>alert(1)</script></html>') });
+ assert.strictEqual(r.status, 200);
+ r = await call('POST', `${O}/${fake}/complete`);
+ assert.deepStrictEqual([r.status, r.body.code], [415, 'media.object.content_mismatch'], 'bytes that are not the declared image');
+ assert.strictEqual(model.getObject(fake).lifecycle_status, 'uploading', 'the object stays uploading');
+ r = await call('POST', O, { body: { kind: 'screenshot' } });
+ r = await call('PUT', `${O}/${r.body.id}/content`, { raw: Buffer.from('<svg/>'), headers: { 'content-type': 'image/svg+xml' } });
+ assert.deepStrictEqual([r.status, r.body.code], [415, 'media.object.unsupported_type'], 'an undeclared type comes from the PUT and is checked too');
+ r = await call('POST', O, { body: { kind: 'file', mime_type: 'image/gif' } });
+ const gifId = r.body.id;
+ await call('PUT', `${O}/${gifId}/content`, { raw: Buffer.from('%PDF-1.4 not a gif') });
+ r = await call('POST', `${O}/${gifId}/complete`);
+ assert.deepStrictEqual([r.status, r.body.code], [415, 'media.object.content_mismatch'], 'inline-served types are checked for any kind');
+ r = await call('POST', O, { body: { kind: 'file', mime_type: 'application/zip' } });
+ const zipId = r.body.id;
+ await call('PUT', `${O}/${zipId}/content`, { raw: Buffer.from('anything at all') });
+ assert.strictEqual((await call('POST', `${O}/${zipId}/complete`)).status, 200, 'attachments are taken as declared');
+ console.log('✅ content types: kind rules at init and PUT, byte checks at complete for inline-served types');
+
+ // ── 3. Multipart ──
+ const PART = 512 * 1024;
+ const big = crypto.randomBytes(PART * 2 + 251424);
+ r = await call('POST', O, { body: { kind: 'file', mime_type: 'application/octet-stream', size_bytes: big.length } });
+ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.object.too_large'], 'over the single-part limit without multipart');
+ assert.match(r.body.detail, /multipart: true/);
+ r = await call('POST', O, { body: { kind: 'file', size_bytes: 5 * 1024 * 1024, multipart: true } });
+ assert.strictEqual(r.status, 413, 'MEDIA_MULTIPART_MAX_MB');
+ r = await call('POST', '/api/v2/tiny/objects', { bearer: 'tiny-key', body: { kind: 'file', size_bytes: 2 * 1024 * 1024, multipart: true, part_size: PART } });
+ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.quota.exceeded'], 'the tenant quota at init');
+ r = await call('POST', O, { body: { kind: 'file', mime_type: 'application/octet-stream', size_bytes: big.length, multipart: true, part_size: PART, filename: 'big.bin', content_hash: sha(big) } });
+ assert.strictEqual(r.status, 201, JSON.stringify(r.body));
+ const mp = r.body;
+ const up = mp.upload.multipart;
+ assert.deepStrictEqual([mp.upload.method, mp.upload.url, up.parts_expected, up.part_size, up.total_size, up.status], ['multipart', null, 3, PART, big.length, 'active']);
+ assert.ok(/^mup_/.test(up.upload_id) && /^mp1\./.test(up.token));
+ const partUrl = (n) => local(up.part_url_template.replace('{part_number}', String(n)));
+ const slice = (n) => big.subarray((n - 1) * PART, Math.min(big.length, n * PART));
+
+ r = await call('PUT', partUrl(1), { bearer: null, raw: slice(1) });
+ assert.deepStrictEqual([r.status, r.body.part_number, r.body.sha256], [200, 1, sha(slice(1))]);
+ r = await call('PUT', `${O}/${mp.id}/content`, { raw: big.subarray(0, 10) });
+ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.object.too_large'], 'no single PUT for a multipart-sized object');
+ r = await call('PUT', partUrl(2), { bearer: null, raw: slice(2).subarray(0, 1000) });
+ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.upload.part_size_mismatch'], 'every part has an exact size');
+ r = await call('PUT', partUrl(2), { bearer: null, raw: slice(2), headers: { 'x-content-sha256': sha(Buffer.from('other')) } });
+ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.upload.part_hash_mismatch']);
+ r = await call('PUT', partUrl(4), { bearer: null, raw: slice(3) });
+ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.upload.invalid_part']);
+ r = await call('PUT', partUrl(2).replace(/token=[^&]+/, 'token=mp1.99999999999.forged'), { bearer: null, raw: slice(2) });
+ assert.strictEqual(r.status, 401);
+
+ // A dropped connection mid-part: nothing of it is kept; the session says what is missing.
+ await new Promise((resolve) => {
+ const u = new URL(partUrl(2));
+ const req = http.request({ host: u.hostname, port: u.port, path: u.pathname + u.search, method: 'PUT', headers: { 'content-length': slice(2).length } });
+ req.on('error', () => resolve());
+ req.write(slice(2).subarray(0, 200000));
+ setTimeout(() => { req.destroy(); setTimeout(resolve, 150); }, 100);
+ });
+ r = await call('GET', local(up.status_url), { bearer: null });
+ assert.deepStrictEqual([r.status, r.body.parts.map((p) => p.part_number), r.body.missing, r.body.received_bytes], [200, [1], [2, 3], PART]);
+ assert.deepStrictEqual(fs.readdirSync(path.join(process.env.OBJECTS_PATH, '.parts', up.upload_id)), ['1'], 'no half part on disk');
+ // Resume: send what is missing (part 3 with the app key instead of the session token).
+ r = await call('PUT', partUrl(2), { bearer: null, raw: slice(2), headers: { 'x-content-sha256': sha(slice(2)) } });
+ assert.strictEqual(r.status, 200);
+ r = await call('PUT', `${O}/${mp.id}/multipart/${up.upload_id}/parts/3`, { raw: slice(3) });
+ assert.strictEqual(r.status, 200, 'the tenant credential works as well as the session token');
+ r = await call('GET', `${O}/${mp.id}/multipart/${up.upload_id}`);
+ assert.deepStrictEqual([r.body.missing, r.body.received_bytes], [[], big.length]);
+
+ r = await call('POST', local(up.complete_url), { bearer: null, body: { parts: [{ part_number: 1, sha256: sha(slice(2)) }] } });
+ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.upload.part_hash_mismatch'], "the client's part list must match");
+ assert.strictEqual(multipart.getSession(up.upload_id).status, 'active', 'and the session stays open');
+ r = await call('POST', local(up.complete_url), { bearer: null, body: { parts: [1, 2, 3].map((n) => ({ part_number: n, sha256: sha(slice(n)) })) } });
+ assert.strictEqual(r.status, 200, JSON.stringify(r.body));
+ assert.deepStrictEqual([r.body.lifecycle_status, r.body.size_bytes, r.body.content_hash], ['ready', big.length, sha(big)]);
+ assert.ok(!fs.existsSync(path.join(process.env.OBJECTS_PATH, '.parts', up.upload_id)), 'parts removed');
+ assert.strictEqual(multipart.getSession(up.upload_id).status, 'completed');
+ assert.deepStrictEqual(fs.readFileSync(model.objectFilePath(model.getObject(mp.id))), big, 'the assembled object is the original');
+ r = await call('GET', `${O}/${mp.id}/download`);
+ r = await call('GET', local(r.body.url), { bearer: null });
+ assert.deepStrictEqual([r.status, sha(Buffer.from(await (await fetch(local((await call('GET', `${O}/${mp.id}/download`)).body.url))).arrayBuffer()))], [200, sha(big)]);
+ r = await call('POST', local(up.complete_url), { bearer: null });
+ assert.strictEqual(r.status, 409, 'completing twice');
+ console.log('✅ multipart: exact parts, sha256 per part, resumable after a dropped connection, assembled and verified');
+
+ // Declare the size later; abort; a session for another object does not open this one; expiry.
+ r = await call('POST', O, { body: { kind: 'file' } });
+ const later = r.body.id;
+ r = await call('POST', `${O}/${later}/multipart`, { body: { part_size: PART } });
+ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.object.invalid'], 'a size is needed');
+ r = await call('POST', `${O}/${later}/multipart`, { body: { size_bytes: PART + 10, part_size: PART } });
+ assert.strictEqual(r.status, 201);
+ const s2 = r.body;
+ r = await call('PUT', `${O}/${later}/content`, { raw: Buffer.alloc(PART + 10) });
+ assert.deepStrictEqual([r.status, r.body.code], [409, 'media.upload.multipart_active']);
+ r = await call('PUT', local(s2.part_url_template.replace('{part_number}', '1')).replace(later, a.id).replace(s2.upload_id, up.upload_id), { bearer: null, raw: Buffer.alloc(PART) });
+ assert.strictEqual(r.status, 401, "a session token opens only its own session's object");
+ r = await call('PUT', local(s2.part_url_template.replace('{part_number}', '1')), { bearer: null, raw: Buffer.alloc(PART) });
+ assert.strictEqual(r.status, 200);
+ r = await call('DELETE', local(s2.abort_url), { bearer: null });
+ assert.deepStrictEqual([r.status, r.body.status], [200, 'aborted']);
+ assert.ok(!fs.existsSync(path.join(process.env.OBJECTS_PATH, '.parts', s2.upload_id)));
+ assert.strictEqual(model.getObject(later).lifecycle_status, 'uploading', 'the object can be sent again');
+ r = await call('PUT', local(s2.part_url_template.replace('{part_number}', '2')), { bearer: null, raw: Buffer.alloc(10) });
+ assert.deepStrictEqual([r.status, r.body.code], [409, 'media.upload.not_active']);
+ r = await call('POST', `${O}/${later}/multipart`, { body: { part_size: PART } });
+ const s3 = r.body;
+ await call('PUT', local(s3.part_url_template.replace('{part_number}', '1')), { bearer: null, raw: Buffer.alloc(PART) });
+ db.run("UPDATE media_uploads SET expires_at = datetime('now', '-1 minute') WHERE id = ?", [s3.upload_id]);
+ fs.mkdirSync(path.join(process.env.OBJECTS_PATH, '.parts', 'mup_orphan'), { recursive: true });
+ const rec = await require('../server/objects/reconcile').reconcile({ appId: 'live' });
+ assert.deepStrictEqual(rec.issues.incomplete_multipart.items.map((i) => [i.upload_id, i.parts_received]), [[s3.upload_id, 1]], 'reconciliation reports an open session past its expiry');
+ assert.deepStrictEqual(multipart.purgeExpired(), { expired: 1, orphan_dirs: 1 });
+ assert.strictEqual(multipart.getSession(s3.upload_id).status, 'expired');
+ assert.deepStrictEqual(fs.readdirSync(path.join(process.env.OBJECTS_PATH, '.parts')), [], 'expired and orphan parts removed');
+ console.log('✅ multipart: size declared later, abort, per-session tokens, single PUT refused while open, expiry purge');
+
+ server.close();
+ console.log('\nobjects multipart + presigned: all checks passed');
+ process.exit(0);
+})().catch((err) => { console.error(err); process.exit(1); });
new file mode 100644
new file mode 100644
@@ -0,0 +1,187 @@
@@ -0,0 +1,187 @@
+'use strict';
+// scripts/r2-eviction-drill.js against a fake S3 (B2 and R2 buckets on one local endpoint, spoken to by
+// the real AWS SDK through vod-storage) and a local Media serving /v/:id: the dry run changes nothing;
+// the drill refuses when the canonical copy looks wrong, when a hold freezes placement, or when there
+// is no R2 copy; --execute evicts the R2 copy, proves /v/:id is served from B2 with the same bytes,
+// re-warms R2 and proves it again; a failed re-warm leaves the VOD served from B2; the CLI writes the artifact.
+const assert = require('assert');
+const fs = require('fs');
+const os = require('os');
+const path = require('path');
+const http = require('http');
+const crypto = require('crypto');
+const { execFile } = require('child_process');
+// The fake S3 lives in this process, so the CLI must run without blocking it (no spawnSync).
+const runCli = (args) => new Promise((resolve) => {
+ execFile(process.execPath, [path.join(__dirname, '../scripts/r2-eviction-drill.js'), ...args], { env: process.env, encoding: 'utf8' },
+ (err, stdout, stderr) => resolve({ status: err ? err.code : 0, stdout, stderr }));
+});
+
+const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-r2drill-'));
+const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+
+// ── Fake S3: path-style /<bucket>/<key>; HEAD, GET (Range), PUT, DELETE ──
+const buckets = { 'b2-bucket': new Map(), 'r2-bucket': new Map() };
+const s3log = [];
+let refusePutTo = null;
+function decodeAwsChunked(buf) {
+ const out = [];
+ let i = 0;
+ for (;;) {
+ const nl = buf.indexOf('\r\n', i);
+ const size = parseInt(buf.subarray(i, nl).toString().split(';')[0], 16);
+ if (!size) break;
+ out.push(buf.subarray(nl + 2, nl + 2 + size));
+ i = nl + 2 + size + 2;
+ }
+ return Buffer.concat(out);
+}
+const s3 = http.createServer((req, res) => {
+ const chunks = [];
+ req.on('data', (c) => chunks.push(c));
+ req.on('end', () => {
+ const u = new URL(req.url, 'http://x');
+ const [, bucket, ...rest] = decodeURIComponent(u.pathname).split('/');
+ const key = rest.join('/');
+ const store = buckets[bucket];
+ s3log.push([req.method, bucket, key]);
+ if (!store) { res.writeHead(404); return res.end(); }
+ const obj = store.get(key);
+ if (req.method === 'HEAD' || req.method === 'GET') {
+ if (!obj) { res.writeHead(404, { 'Content-Type': 'application/xml' }); return res.end(req.method === 'GET' ? '<Error><Code>NoSuchKey</Code></Error>' : undefined); }
+ const m = /bytes=(\d+)-(\d+)?/.exec(req.headers.range || '');
+ if (m && req.method === 'GET') {
+ const start = Number(m[1]), end = Math.min(obj.length - 1, m[2] != null ? Number(m[2]) : obj.length - 1);
+ res.writeHead(206, { 'Content-Length': end - start + 1, 'Content-Range': `bytes ${start}-${end}/${obj.length}`, ETag: '"e"' });
+ return res.end(obj.subarray(start, end + 1));
+ }
+ res.writeHead(200, { 'Content-Length': obj.length, ETag: '"e"', 'Content-Type': 'video/webm' });
+ return res.end(req.method === 'GET' ? obj : undefined);
+ }
+ if (req.method === 'PUT') {
+ if (refusePutTo === bucket) { res.writeHead(500, { 'Content-Type': 'application/xml' }); return res.end('<Error><Code>InternalError</Code></Error>'); }
+ let body = Buffer.concat(chunks);
+ if (req.headers['x-amz-decoded-content-length'] || /aws-chunked/.test(req.headers['content-encoding'] || '')) body = decodeAwsChunked(body);
+ store.set(key, body);
+ res.writeHead(200, { ETag: '"e"' });
+ return res.end();
+ }
+ if (req.method === 'DELETE') { store.delete(key); res.writeHead(204); return res.end(); }
+ res.writeHead(405); res.end();
+ });
+});
+
+(async () => {
+ await new Promise((r) => s3.listen(0, '127.0.0.1', r));
+ const endpoint = `http://127.0.0.1:${s3.address().port}`;
+ Object.assign(process.env, {
+ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
+ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'),
+ MEDIA_B2_ENDPOINT: endpoint, MEDIA_B2_BUCKET: 'b2-bucket', MEDIA_B2_KEY_ID: 'k', MEDIA_B2_APP_KEY: 's', MEDIA_B2_REGION: 'us-west-004',
+ MEDIA_R2_ENDPOINT: endpoint, MEDIA_R2_BUCKET: 'r2-bucket', MEDIA_R2_ACCESS_KEY_ID: 'k', MEDIA_R2_SECRET_ACCESS_KEY: 's',
+ AWS_REQUEST_CHECKSUM_CALCULATION: 'WHEN_REQUIRED', AWS_RESPONSE_CHECKSUM_VALIDATION: 'WHEN_REQUIRED',
+ });
+ const db = require('../server/db/database');
+ const model = require('../server/objects/model');
+ const storage = require('../server/vod/vod-storage');
+ const { runDrill, providerOfUrl } = require('../scripts/r2-eviction-drill');
+ const conn = db.getDb();
+ require('../server/views/service').ensureSchema();
+
+ // Media serving /v/:id, as production does.
+ const express = require('express');
+ const app = express();
+ app.use('/', require('../server/public/routes'));
+ const media = http.createServer(app);
+ await new Promise((r) => media.listen(0, '127.0.0.1', r));
+ const baseUrl = `http://127.0.0.1:${media.address().port}`;
+
+ const bytes = crypto.randomBytes(3 * 1024 * 1024 + 123);
+ const addVod = (id, { provider = 'r2', visibility = 'public', size = bytes.length } = {}) => {
+ const key = `vods/drill-${id}.webm`;
+ conn.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, file_size, is_public, visibility, duration_seconds, storage_provider, storage_key)
+ VALUES (?, 'live', 5, 'Drill', ?, ?, ?, ?, 60, ?, ?)`).run(id, `/gone/drill-${id}.webm`, size, visibility === 'private' ? 0 : 1, visibility, provider, key);
+ buckets['b2-bucket'].set(key, bytes);
+ if (provider === 'r2') buckets['r2-bucket'].set(key, bytes);
+ model.sync('vod', id);
+ return key;
+ };
+ const deps = { storage, db, model };
+ const opts = (o) => ({ http: true, baseUrl, maxMb: 512, ...o });
+ assert.strictEqual(providerOfUrl(`${endpoint}/r2-bucket/vods/x.webm?X-Amz-Signature=1`, storage), 'r2');
+ assert.strictEqual(providerOfUrl('https://b2-bucket.s3.example/vods/x.webm', { bucketFor: (p) => `${p}-bucket`, endpointFor: () => 'https://s3.example' }), 'b2');
+
+ // ── Dry run: checks only ──
+ const key1 = addVod(1);
+ let art = await runDrill(opts({ vodId: 1 }), deps);
+ assert.strictEqual(art.verdict, 'dry-run', art.reason);
+ assert.deepStrictEqual(art.steps.map((s) => s.name), ['plan']);
+ assert.ok(buckets['r2-bucket'].has(key1) && db.get('SELECT storage_provider FROM vods WHERE id = 1').storage_provider === 'r2', 'nothing moved');
+ assert.ok(!s3log.some(([m]) => m === 'PUT' || m === 'DELETE'), 'no writes to either bucket');
+ assert.strictEqual(art.checks.first_mib.b2, art.checks.first_mib.r2);
+ console.log('✅ dry run: preconditions and copy checks only, nothing changes');
+
+ // ── Refusals ──
+ const key2 = addVod(2);
+ buckets['b2-bucket'].set(key2, bytes.subarray(0, 100));
+ assert.match((await runDrill(opts({ vodId: 2, execute: true }), deps)).reason, /differ in size/);
+ buckets['b2-bucket'].set(key2, Buffer.concat([Buffer.from('X'), bytes.subarray(1)]));
+ assert.match((await runDrill(opts({ vodId: 2, execute: true }), deps)).reason, /first MiB/);
+ buckets['b2-bucket'].delete(key2);
+ assert.match((await runDrill(opts({ vodId: 2, execute: true }), deps)).reason, /B2 canonical copy is missing/);
+ assert.ok(buckets['r2-bucket'].has(key2), 'a refused drill deletes nothing');
+ addVod(3, { provider: 'b2' });
+ assert.match((await runDrill(opts({ vodId: 3, execute: true }), deps)).reason, /not R2/);
+ addVod(4);
+ const hold = model.placeHold({ object_id: db.get('SELECT object_id FROM vods WHERE id = 4').object_id, kind: 'evidence', reason: 'drill test' });
+ assert.match((await runDrill(opts({ vodId: 4, execute: true }), deps)).reason, /retention hold/);
+ model.releaseHold(hold.id, 'test');
+ addVod(5, { visibility: 'private' });
+ assert.match((await runDrill(opts({ vodId: 5, execute: true }), deps)).reason, /private/);
+ assert.strictEqual((await runDrill(opts({ pick: true }), deps)).vod_id, 1, '--pick takes the smallest public R2 VOD that is not held');
+ console.log('✅ refused: canonical copy wrong size / different bytes / missing, not on R2, held, private over HTTP');
+
+ // ── Execute: evict, served from B2, re-warm, served from R2 ──
+ s3log.length = 0;
+ art = await runDrill(opts({ vodId: 1, execute: true }), deps);
+ assert.strictEqual(art.verdict, 'pass', `${art.reason} ${JSON.stringify(art.steps, null, 1)}`);
+ assert.deepStrictEqual(art.steps.map((s) => [s.name.split(':')[0], s.ok]), [['before', true], ['evict', true], ['from B2', true], ['re-warm', true], ['after', true]]);
+ assert.deepStrictEqual([art.steps[0].provider, art.steps[2].provider, art.steps[4].provider], ['r2', 'b2', 'r2'], 'GET /v/1 went R2 -> B2 -> R2');
+ assert.deepStrictEqual([art.steps[0].status, art.steps[2].range_status], [302, 206]);
+ const sha = crypto.createHash('sha256').update(bytes.subarray(0, 1024 * 1024)).digest('hex');
+ assert.ok(art.steps.filter((s) => s.first_mib_sha256).every((s) => s.first_mib_sha256 === sha), 'the same bytes every time');
+ assert.ok(s3log.some(([m, b]) => m === 'DELETE' && b === 'r2-bucket') && s3log.some(([m, b]) => m === 'PUT' && b === 'r2-bucket'), 'the R2 copy was really deleted and written back');
+ assert.ok(!s3log.some(([m, b]) => (m === 'DELETE' || m === 'PUT') && b === 'b2-bucket'), 'B2 was never written');
+ assert.deepStrictEqual(buckets['r2-bucket'].get(key1), bytes, 'R2 holds the same bytes again');
+ assert.strictEqual(db.get('SELECT storage_provider FROM vods WHERE id = 1').storage_provider, 'r2');
+ assert.deepStrictEqual(art.locations.after_evict.map((l) => l.provider).sort(), ['b2']);
+ assert.deepStrictEqual(art.locations.after.map((l) => [l.provider, l.state]).sort(), [['b2', 'present'], ['r2', 'present']]);
+ console.log('✅ execute: R2 copy evicted, /v served from B2 with the same bytes, R2 re-warmed and serving again');
+
+ // ── A failed re-warm leaves the VOD served from B2 ──
+ addVod(6);
+ refusePutTo = 'r2-bucket';
+ art = await runDrill(opts({ vodId: 6, execute: true, http: false }), deps);
+ refusePutTo = null;
+ assert.strictEqual(art.verdict, 'fail');
+ assert.deepStrictEqual(art.steps.map((s) => s.ok), [true, true, true, false]);
+ assert.strictEqual(art.steps[0].via, 'resolvePlayback', '--no-http decides in-process');
+ assert.strictEqual(db.get('SELECT storage_provider FROM vods WHERE id = 6').storage_provider, 'b2');
+ assert.strictEqual((await storage.resolvePlayback(db.get('SELECT * FROM vods WHERE id = 6'))).provider, 'b2', 'still served, from B2');
+ console.log('✅ a failed re-warm stops the drill with the VOD served from B2');
+
+ // ── The CLI writes the artifact ──
+ const out = path.join(tmp, 'artifact.json');
+ const cli = await runCli(['--vod', '1', '--no-http', '--out', out]);
+ assert.strictEqual(cli.status, 0, cli.stderr + cli.stdout);
+ const saved = JSON.parse(fs.readFileSync(out, 'utf8'));
+ assert.deepStrictEqual([saved.drill, saved.mode, saved.verdict, saved.vod_id], ['r2-eviction', 'dry-run', 'dry-run', 1]);
+ const refused = await runCli(['--vod', '3', '--out', path.join(tmp, 'r.json')]);
+ assert.strictEqual(refused.status, 2, 'refused -> exit 2');
+ console.log('✅ CLI: dry run by default, JSON artifact written, exit codes');
+
+ media.close();
+ s3.close();
+ console.log('\nr2 eviction drill: all checks passed');
+ process.exit(0);
+})().catch((err) => { console.error(err); process.exit(1); });