openvibers/openvibe.media Commits
Canonical object model (Wave 4): media_objects with locations, relationships, variants, jobs, retention holds and the public-size invariant; vods/clips/files/screenshot pastes become projections via object_id
Changes
@@@ -23,7 +23,8 @@ server/config.js env config (PORT, DB_PATH, *_PATH, MEDIA_B2_*/R2_*, RTP pool, OV_NETWORK_URL)auth.js tenancy + auth middleware, JWKS fetch, app seedingwebhooks.js HMAC-signed outbound webhooks-- db/schema.sql apps, vods, clips, pastes(+likes/comments), files, content_views, media_settings++ db/schema.sql apps, vods, clips, pastes(+likes/comments), files, content_views, media_settings,++ media_objects/locations/relationships/variants/jobs/holds/invariant_violationsdb/database.js better-sqlite3 helpers, all app_id-scoped; legacy row importervod/recorder.js ffmpeg recording: RTMP pull + RTP (SDP) ingest, codec passthroughvod/media-tools.js probes, seekable remux, DVR sidecar, chunk-segment concat@@@ -39,8 +40,11 @@ server/admin/routes.js /api/v1/:app/admin/storage (disk, tiers, buckets, bulk ops)thumbnails/ thumbnail service + /api/v1/:app/thumbnailspublic/routes.js public /v /c /p /t /f++ objects/ canonical object model: model (projections, holds), routes (/api/v2 + /o),++ backfill, reconcile, invariant, signing — see docs/object-model.mdvendor/openvibe-shared/ vendored shared helpers (do not edit; re-sync from canonical)scripts/smoke-test.sh end-to-end smoke test (boots a temp instance)++scripts/backfill-objects.js / reconcile-objects.js / object-invariant.js object-model operator tools```## Visitor sign-in@@@ -192,7 +196,7 @@ directories are shared across apps) and responses carry a `note` saying so.### Webhooks (outbound)`POST` to the app's `webhook_url` with body--`{ "event": "vod.ready"|"vod.failed"|"clip.ready"|"clip.failed", "app_id", "data" }`++`{ "event": "vod.ready"|"vod.failed"|"clip.ready"|"clip.failed"|"media.object.uploaded", "app_id", "data" }`and header `X-OVMedia-Signature: sha256=<hex hmac-sha256 of the raw body withthe app's webhook_secret>`. 3 attempts with backoff, 10 s timeout.@@@ -216,6 +220,42 @@ the app's webhook_secret>`. 3 attempts with backoff, 10 s timeout.Private items respond 403/404 unless the request bears the owning app's APIkey or the owning user's JWT.++`GET /o/:id` serves object bytes by canonical id (`med_…`): public/unlisted++objects openly, private ones only with a valid signature from++`/api/v2/:app/objects/:id/download`, deleted ones 410 (see below).++++## Object model and API v2++++Every stored blob is a **media object** (`med_<ULID>`, `media_objects`) with one++`media_locations` row per copy (local / B2 canonical / R2 cache, each `present`,++`missing`, `pending` or `corrupt`). The `vods`, `clips`, `files` and screenshot++`pastes` rows are typed projections over objects (`object_id` column); every++existing route and response is unchanged, and the old write paths keep the++model current. Full reference: **[docs/object-model.md](docs/object-model.md)**.++++- **Backfill** — `node scripts/backfill-objects.js [--dry-run]`: one object per++ vod, clip, file, screenshot, avatar and thumbnail; idempotent; never moves++ bytes or calls B2/R2 (remote copies stay `pending`). The service runs the++ `--only-missing` form 15 s after boot.++- **API** — `/api/v2/:app/objects`: init → `PUT /:id/content` (sha256, size,++ quota) → `/:id/complete`; `GET /:id`, cursor `GET /`, soft `DELETE /:id`++ (bytes kept `MEDIA_DELETE_RETENTION_DAYS`), `/:id/restore`, `/:id/download`++ (302 for public, HMAC-signed short-lived URL for private —++ `MEDIA_SIGNING_SECRET`). App key, or a Network service token with++ `media.object.upload` / `media.object.read` for namespace `:app`;++ `X-OV-Subject` sets the owner.++- **Retention holds** — `media_holds` (`moderation`, `dmca`, `creator_pin`,++ `admin`, `evidence`; `/:id/holds`, app key only): a held object cannot be++ deleted by any path (409 / DB trigger) or moved between tiers.++- **Reconciliation** — `node scripts/reconcile-objects.js [--verify] [--hash]`:++ read-only by default; `--verify` HEADs B2/R2 and records location states.++ Reports missing canonical copies, lost local files, size/hash mismatches,++ orphan locations, deleted-but-still-served objects and unprojected rows.++- **Public object-size invariant** — `MEDIA_PUBLIC_OBJECT_MAX_MB` (500) /++ `_TARGET_MB` (256) / `_WARN_MB` (384): v2 refuses oversized public playback++ uploads; `node scripts/object-invariant.js` lists offenders and records them in++ `media_invariant_violations`. No automatic re-encoding.++## Storage tiering`vod/vod-storage.js` (ported nearly as-is): local → B2 canonical cold tier →nnew file mode 100644@@@ -0,0 +1,258 @@++# Object model++++Roadmap Wave 4 turns Media from predecessor-shaped VOD/clip/file tables into one canonical object++platform. Every stored blob is a **media object** with an id `med_<ULID>`. The inherited++`vods`, `clips`, `files` and screenshot `pastes` rows stay as they are, and each one is now a typed++projection over an object through its `object_id` column. Every existing URL and v1 response is++unchanged. New code talks to objects through the [v2 API](#object-api-v2).++++Code: `server/objects/` (`model.js`, `routes.js`, `backfill.js`, `reconcile.js`, `invariant.js`,++`signing.js`). Schema: the bottom of `server/db/schema.sql`. Tests: `test/objects-*.test.js`.++++## Tables++++| table | what it holds |++|---|---|++| `media_objects` | `id` (med_…), `app_id` (tenant), `namespace` (capability namespace; = app today), `kind` (`vod` `clip` `file` `thumbnail` `screenshot` `avatar` `asset`), `owner_subject` (usr_… when known), `owner_app` + `owner_user_id` (legacy owner in the app's own user-id space), `visibility` (`public` `unlisted` `private`), `lifecycle_status` (`uploading` `ready` `failed` `archived` `deleted`), `mime_type`, `size_bytes`, `content_hash` (sha256), `canonical_provider` + `canonical_key`, `legacy_ref`, `metadata` (JSON), `created_at` / `updated_at` / `deleted_at` |++| `media_locations` | one row per provider copy (`UNIQUE(object_id, provider)`): `provider` (`local` `b2` `r2`), `bucket`, `key` (absolute path for local, object key for B2/R2), `storage_class` (`hot` local, `cold` B2, `cache` R2), `state`, `checksum`, `size_bytes`, `verified_at` |++| `media_relationships` | `from_object_id`, `relation`, `to_object_id`, `metadata`. In use: `clip_of` (clip to vod, with start/end), `thumbnail_of` (thumbnail to vod/clip). Reserved: `derived_from`, `screenshot_of` |++| `media_variants` | `object_id`, `variant_name`, `derived_object_id`, `recipe`. In use: `thumbnail` |++| `media_jobs` | generic derivative/maintenance jobs (`job_type`, `status`, `attempts`, `checkpoint`, `error`). **Schema only in this pass:** nothing enqueues or consumes jobs yet |++| `media_holds` | retention holds, see [Holds](#retention-holds) |++| `media_invariant_violations` | public playback objects over the size policy, see [Invariant](#public-object-size-invariant) |++++Location `state`:++++- `present`: verified. The local file exists, or a HEAD answered with the right size.++- `missing`: verified absent.++- `pending`: believed there but not verified yet. Every remote copy starts here until reconciliation runs with `--verify`.++- `corrupt`: the size or hash does not match.++++A copy that reconciliation has verified keeps that state when the row is projected again. The++state resets only when the key changes.++++### Ids and legacy refs++++Ids come from `openvibe-contracts` `ids.newId('media')`. A backfilled object takes its ULID time++from the original row's `created_at`, so ids sort by creation time. Projected objects also carry++the transitional `media.media-ref@1` legacy form in `legacy_ref`, which is unique:++++| row | `kind` | `legacy_ref` |++|---|---|---|++| `vods.id` | vod | `legacy:<app>:vod:<id>` |++| `clips.id` | clip | `legacy:<app>:clip:<id>` |++| `files.key` | file | `legacy:<app>:file:<key>` |++| screenshot paste | screenshot | `legacy:<app>:paste:<slug>` |++| avatar (screenshot paste with `metadata.kind = 'avatar'`) | avatar | `legacy:<app>:avatar:<slug>` |++| `vods`/`clips.thumbnail_url` (`/t/<name>`) | thumbnail | `legacy:<app>:thumbnail:<name>` |++++Every v2 route that takes `:id` accepts either form. Native v2 objects have `legacy_ref = NULL`.++++### How a row maps to its object++++`model.sync(kind, id)` derives the object and its locations from the row as it is at that moment.++The backfill and every write hook use the same function.++++- **vod / clip.**++ - Local copy: the file under `VOD_PATH` (vods, resolved by basename as playback does) or the clip's own path.++ - B2 copy (canonical, key `vods/<basename>` or `storage_key`): added when `storage_provider` is `b2`/`r2` or a `storage_key` is set.++ - R2 copy (cache): added when the provider is `r2`.++ - Canonical copy: B2 when one exists, otherwise local.++ - Lifecycle: recording → `uploading`; `corrupt`/`zero_byte`/`missing_file` → `failed`; `ready` otherwise. Clips: `processing` → `uploading`, `failed` → `failed`.++ - Visibility: `visibility`, falling back to `is_public`, the same way playback decides.++ - Clips-only (ephemeral) recordings are never projected.++- **file.** The local file under `FILES_PATH/<app>/<key>`. `visibility = public`, because `/f/:key` serves every file without auth. The row's sha256 becomes both `content_hash` and the location `checksum`.++- **screenshot / avatar.** `screenshot_path`. Text pastes have no bytes and no object; paste text belongs to Community.++- **thumbnail.** One object per vod/clip, reached through the parent's `thumbnail` variant and **updated in place** when the picture is regenerated (live recordings refresh it every couple of minutes). Visibility is `public` when the parent is public and `unlisted` otherwise, because `/t/<name>` serves anyone who has the name. An external `thumbnail_url` is skipped.++++## Backfill++++```++node scripts/backfill-objects.js [--dry-run] [--only-missing] [--json] [--out report.json] [--db ./data/media.db]++```++++The backfill creates one object per vod, clip, file, screenshot/avatar paste and thumbnail, with++relationships and thumbnail variants.++++- **Idempotent.** Objects are keyed by `legacy_ref`, so a second run updates them in place and creates nothing.++- **Never moves or deletes bytes, and never calls B2/R2.** A local copy is `present` when the file exists and `missing` when it does not. Remote copies stay `pending` until `reconcile-objects.js --verify` checks them.++- **One transaction,** with one savepoint per row: a failing row is reported and leaves nothing half-written. `--dry-run` rolls everything back and only reports.++- **Report:** counts per kind (seen, created, updated, skipped), location states, skipped rows with a reason, and errors. A real run stores the report in `media_settings` under `objects.backfill.last_report`.++- **At boot:** the service runs the `--only-missing` form itself 15 s after start, for rows with no `object_id`. The first boot after the upgrade therefore projects everything.++++Skip reasons today: `clips-only recording (ephemeral, never published)`, `screenshot paste without a file path`, `external thumbnail url`.++++The run holds SQLite's write lock for its duration, typically seconds. Prefer the boot backfill, or++run the script while traffic is quiet.++++### Keeping the model current++++The old write paths re-project their row after they change it:++++| path | hook |++|---|---|++| vod create | `POST /vods` |++| recording start | `recorder` sets `file_path` |++| every finalize outcome | `finalize.finalizeVod` |++| visibility | `db.setVodVisibility` / `setClipVisibility` |++| health | `db.updateVodHealth` |++| title | `PUT /vods/:id`, `PUT /clips/:id` |++| clip create | `db.createClip` (API cuts, uploads and auto-clips) |++| clip cut / re-cut result | clips routes and `clip-jobs` |++| file upload | `db.createFile` |++| thumbnail generate / upload | `thumbnail-service`, `POST /thumbnails` |++| screenshot upload / censor | pastes routes |++| avatar ingest | `avatars/ingest` |++| tier moves | `moveToCold`, `moveToHot`, `promoteToR2`, `demoteFromR2`, sweep, `migrateLegacy`: the copy the move verified is marked `present` |++++**Deletes need no hook.** A SQLite trigger on `vods`, `clips`, `files` and `pastes` marks the++object `deleted` whenever its row is deleted, which covers all of the inherited delete paths. Hooks++never throw: the legacy write has already happened, so a failed sync only logs a warning. The next++sync or backfill repairs it.++++## Object API v2++++All routes live under `/api/v2/:app/objects`.++++**Auth.** One of:++++- the app's API key;++- a Network service token that grants `media.object.upload` (writes) or `media.object.read` (reads) for the namespace `:app`. `tenantAuth({ capability })` enforces this, as it does on v1.++++**Owner.**++++- `X-OV-Subject: usr_…` (or `user:usr_…`) sets `owner_subject`.++- With an app key, `X-OV-User-Id` acts as one of the app's users. That caller sees public and unlisted objects plus its own, and can only write its own.++++**Errors** are RFC 9457 problem bodies (`openvibe-contracts` `http.sendProblem`).++++| method | path | notes |++|---|---|---|++| POST | `/` | init. `{ kind, visibility (default private), mime_type, size_bytes, filename, content_hash (expected sha256), metadata (≤ 16 KB), user_id }` returns 201 `{ id, object, upload: { method: 'PUT', url, token, expires_at, max_bytes, complete_url } }`. Checks at init: `size_bytes` against `MEDIA_OBJECT_MAX_MB` (413), the quota including reservations (413 `media.quota.exceeded`), and the public-size invariant (422) |++| PUT | `/:id/content` | the bytes, single part. Auth is the upload token (`?token=` from init, or `X-Upload-Token`) or the usual credential. Streams to disk computing sha256. Refuses more than the declared size or the limit (413), a size mismatch (400) and quota overrun (413). Stored at `OBJECTS_PATH/<app>/<id>` with a verified local location. Mounted ahead of the JSON body parser, so any Content-Type is taken as raw bytes; the Content-Type becomes `mime_type` when init set none. Can be repeated while the object is `uploading` |++| POST | `/:id/complete` | checks the expected hash (422 `media.object.hash_mismatch`), the invariant and the quota. Sets `ready` and sends the `media.object.uploaded` webhook. Also accepts the upload token |++| GET | `/:id` | metadata: providers and states of each copy, never paths or keys |++| GET | `/` | cursor list, newest first. `?limit (≤200)&cursor&kind&visibility&status&owner (usr_…)&user_id&include_deleted` returns `{ objects, next_cursor }` |++| DELETE | `/:id` | soft delete: `lifecycle_status = deleted`, and the bytes are kept for `MEDIA_DELETE_RETENTION_DAYS` (default 30). 409 `media.object.held` under a hold. 409 `media.object.legacy_managed` for projected objects, which are deleted through their v1 route |++| POST | `/:id/restore` | undo a soft delete within the retention period (410 once purged) |++| GET | `/:id/download` | public or unlisted: 302 to the public location (the inherited URL for projected objects such as `/v/:id`, `/c/:id`, `/f/:key`, `/t/:name`, `/p/:slug/screenshot`; otherwise `/o/:id`). Private: `{ url, expires_at }` signed for `?ttl` seconds (30 to 3600, default `MEDIA_SIGNED_URL_TTL_S` = 300), or a 302 with `?redirect=1`. `?format=json` always answers JSON. 410 when deleted |++| GET | `/:id/holds` | active holds (`?all=1` includes released ones) |++| POST | `/:id/holds` | **app key only.** `{ kind, reason, created_by }` |++| DELETE | `/:id/holds/:holdId` | **app key only.** Releases the hold (`released_at`, `released_by`) |++++**Public bytes: `GET /o/:id`.**++++- Public and unlisted `ready` objects are served openly.++- Private objects need a valid `?exp=&sig=` from `/download`. Without one the answer is 404, so a private object cannot be told apart from a missing one.++- A deleted object answers 410.++- A local copy streams with Range support. Otherwise the route redirects 302 to a presigned R2 URL, then a presigned B2 URL.++- Headers: `X-Content-Type-Options: nosniff` and `X-Robots-Tag: noindex`. Content is `inline` only for images (not SVG), video, audio, PDF and plain text, and `attachment` for everything else.++++**Signing.** HMAC-SHA256 over the purpose (`get` or `put`), the object id and the expiry, keyed with++`MEDIA_SIGNING_SECRET`. A download signature cannot be used as an upload token, or the other way++round. Without the secret, a random per-process secret is used and a warning is logged, so links++stop working at the next restart. **Set `MEDIA_SIGNING_SECRET` in production.**++++**Quota.** `apps.quota_bytes` is compared against v1 file bytes plus native objects that are++`uploading` (their declared size counts as a reservation) or `ready`. Soft-deleted objects do not++count. The v1 files route still counts only v1 files, so its response is unchanged.++++**Purge.** Every hour, the service removes the bytes of native objects whose retention period has++passed and that have no active hold. The object row stays `deleted` with `metadata.purged_at`.++Projected objects are never purged by this job.++++## Retention holds++++`media_holds(object_id, kind, reason, created_by, created_at, released_at, released_by)`. The++`kind` is one of `moderation`, `dmca`, `creator_pin`, `admin`, `evidence`. While any hold is++unreleased:++++- **No delete path works.**++ - v2 `DELETE` answers 409.++ - The v1 deletes for vods, clips and files, and the admin bulk delete, answer 409 or report `held`.++ - Quarantine cleanup skips the object.++ - `deleteVodObjects` keeps the bytes.++ - Paste screenshot removal keeps the file, and censor answers 409.++ - A `BEFORE DELETE` trigger on each projected table, plus a trigger on `media_objects.lifecycle_status`, refuses the change, so paths nobody has hooked are still covered.++- **No tier move.** `moveToCold`, `promoteToR2` and `demoteFromR2` return `{ ok: false, held: true }`, and the sweep counts these as `skippedHeld` instead of errors. A hold freezes an object's placement. `moveToHot` (restoring a copy to local disk) is still allowed.++++## Reconciliation++++```++node scripts/reconcile-objects.js [--verify] [--hash] [--app live] [--json] [--out report.json] [--db ./data/media.db]++```++++**Modes.**++++- **Default: read-only.** Checks local files and database consistency, writes nothing and calls no provider.++- `--verify`: also HEADs every B2/R2 location with the storage engine's S3 client, and records what it finds on `media_locations` (`state`, `size_bytes`, `verified_at`). An unconfigured provider counts as *unverifiable*, never as *missing*.++- `--hash`: computes the sha256 of local files ≤ 512 MB for objects that carry a `content_hash`.++++The exit code is 1 when issues were found.++++**Issues reported** (each with a count and up to 200 examples):++++| issue | meaning |++|---|---|++| `no_canonical_location` | a ready object with no location row for its canonical provider |++| `canonical_missing_replica_present` | the canonical copy is missing or corrupt while another copy is present (for example B2 lost it but R2 has it) |++| `no_present_copy` | a ready object whose every copy is missing or corrupt |++| `missing_local_file` | a ready object's local copy is not on disk |++| `remote_missing` | a ready object's B2/R2 copy answered 404 (`--verify`) |++| `size_mismatch` / `hash_mismatch` | the bytes differ from the object's size or sha256 |++| `orphan_location` | a location row whose object does not exist |++| `deleted_publicly_reachable` | a deleted object whose inherited row still serves it, or a thumbnail of a deleted object that is still served by name |++| `missing_projection` | an inherited row with no object yet (run the backfill) |++++**Not covered yet:** orphan *provider* objects (bucket keys with no row, which needs bucket++listing) and incomplete multipart uploads. `server/objects/reconcile.js` takes a `head` function,++so tests run it against a fake provider.++++## Public object-size invariant++++This is roadmap W4 deliverable 5, stated as **policy** rather than a copied constant:++++| setting | env | default |++|---|---|---|++| max: no public playback object above this | `MEDIA_PUBLIC_OBJECT_MAX_MB` | 500 |++| target | `MEDIA_PUBLIC_OBJECT_TARGET_MB` | 256 |++| warn | `MEDIA_PUBLIC_OBJECT_WARN_MB` | 384 |++++A *public playback object* is a `ready` vod or clip whose visibility is `public` or `unlisted`.++++- **Validator.** `server/objects/invariant.js` classifies a size as `ok`, `above_target`, `warn` or `violation`. The functions are pure.++- **Upload time.** v2 refuses to init or complete a public playback object above the max (422 `media.invariant.public_object_too_large`). Private objects are not playback objects.++- **Violations table.** Every vod/clip sync records warn and violation levels in `media_invariant_violations`, one row per object. The row is resolved when the object shrinks, becomes private or is deleted.++- **Report.**++++ ```++ node scripts/object-invariant.js [--dry-run] [--app live] [--json] [--db ./data/media.db]++ ```++++ Lists every public playback object above target, largest first, records the rows, and resolves rows that no longer apply. `--dry-run` writes nothing.++++**Nothing is re-encoded in this pass.** Most whole-stream VODs are far above 500 MB. They will show++up as violations until the segment-native model lands (roadmap W4 deliverable 7, capture in W7),++which splits playback into objects that fit.++++## Configuration++++| env | default | meaning |++|---|---|---|++| `OBJECTS_PATH` | `./data/objects` | native object bytes (`<app>/<id>`, `.tmp/` while uploading) |++| `MEDIA_OBJECT_MAX_MB` | 256 | single-part upload limit |++| `MEDIA_DELETE_RETENTION_DAYS` | 30 | soft-deleted native bytes are kept this long |++| `MEDIA_SIGNING_SECRET` | *(random per process)* | HMAC key for signed URLs and upload tokens |++| `MEDIA_SIGNED_URL_TTL_S` | 300 | default signed download lifetime |++| `MEDIA_UPLOAD_TOKEN_TTL_S` | 3600 | upload token lifetime |++| `MEDIA_PUBLIC_OBJECT_{MAX,TARGET,WARN}_MB` | 500 / 256 / 384 | invariant thresholds |++++## Not in this pass++++- Multipart uploads and presigned direct-to-bucket uploads. Native bytes are stored locally.++- Tiering of native objects to B2/R2.++- Copy/move aliases, lifecycle rules and an S3-compatible façade.++- A `media_jobs` worker.++- Orphan-bucket-object and incomplete-multipart detection.++- The segment-native timeline.++- App assets (emotes and sounds, the `assets` table) are not projected yet. The `asset` kind exists for v2 uploads.nnew file mode 100755@@@ -0,0 +1,34 @@++#!/usr/bin/env node++'use strict';++/**++ * Project every existing vod, clip, file, screenshot/avatar paste and thumbnail onto the canonical++ * object model (media_objects + media_locations; docs/object-model.md). Idempotent — re-runs update++ * in place. Never moves or deletes bytes and never calls B2/R2 (remote copies stay 'pending' until++ * scripts/reconcile-objects.js --verify).++ *++ * node scripts/backfill-objects.js [--dry-run] [--only-missing] [--json] [--out report.json] [--db ./data/media.db]++ *++ * The service runs the --only-missing form itself shortly after boot. The run is one transaction++ * (a dry run rolls it back), so it holds SQLite's write lock for its duration — seconds, typically.++ */++const fs = require('fs');++const path = require('path');++++const args = process.argv.slice(2);++const has = (name) => args.includes(`--${name}`);++const arg = (name, dflt) => { const i = args.indexOf(`--${name}`); return i >= 0 ? args[i + 1] : dflt; };++if (arg('db')) process.env.DB_PATH = path.resolve(arg('db'));++++const { backfill, summarize } = require('../server/objects/backfill');++const report = backfill({ dryRun: has('dry-run'), onlyMissing: has('only-missing') });++if (arg('out')) fs.writeFileSync(arg('out'), JSON.stringify(report, null, 2));++if (has('json')) console.log(JSON.stringify(report, null, 2));++else {++ console.log(summarize(report));++ const reasons = {};++ for (const s of report.skipped) reasons[s.reason] = (reasons[s.reason] || 0) + 1;++ for (const [r, n] of Object.entries(reasons)) console.log(` skipped ${n}: ${r}`);++ for (const e of report.errors.slice(0, 20)) console.log(` error ${e.table} ${e.id}: ${e.error}`);++}++require('../server/db/database').close();++process.exit(report.errors.length ? 1 : 0);nnew file mode 100755@@@ -0,0 +1,32 @@++#!/usr/bin/env node++'use strict';++/**++ * Public object-size invariant report (roadmap W4 deliverable 5; docs/object-model.md#public-object-size-invariant).++ * Lists public/unlisted ready VOD and clip objects above MEDIA_PUBLIC_OBJECT_TARGET_MB (256), flags++ * those above MEDIA_PUBLIC_OBJECT_WARN_MB (384) and MEDIA_PUBLIC_OBJECT_MAX_MB (500), and records++ * warn/violation rows in media_invariant_violations (resolving rows that no longer apply).++ * Nothing is re-encoded.++ *++ * node scripts/object-invariant.js [--dry-run] [--app live] [--json] [--db ./data/media.db]++ */++const path = require('path');++++const args = process.argv.slice(2);++const has = (name) => args.includes(`--${name}`);++const arg = (name, dflt) => { const i = args.indexOf(`--${name}`); return i >= 0 ? args[i + 1] : dflt; };++if (arg('db')) process.env.DB_PATH = path.resolve(arg('db'));++++const { scan, MB } = require('../server/objects/invariant');++const report = scan({ record: !has('dry-run'), appId: arg('app', null) });++if (has('json')) console.log(JSON.stringify(report, null, 2));++else {++ const t = report.thresholds;++ console.log(`public playback objects — target ${t.targetBytes / MB} MB, warn ${t.warnBytes / MB} MB, max ${t.maxBytes / MB} MB${has('dry-run') ? ' (dry run)' : ''}`);++ console.log(` ok ${report.counts.ok}, above target ${report.counts.above_target}, warn ${report.counts.warn}, violation ${report.counts.violation}`);++ for (const o of report.objects.slice(0, 50)) {++ console.log(` ${o.level.padEnd(12)} ${(o.size_bytes / MB).toFixed(0).padStart(6)} MB ${o.object_id} ${o.legacy_ref || ''}`);++ }++ if (report.objects.length > 50) console.log(` … ${report.objects.length - 50} more (use --json)`);++}++require('../server/db/database').close();++process.exit(0);nnew file mode 100755@@@ -0,0 +1,31 @@++#!/usr/bin/env node++'use strict';++/**++ * Reconcile the object model against the bytes (docs/object-model.md#reconciliation).++ *++ * node scripts/reconcile-objects.js [--verify] [--hash] [--app live] [--json] [--out report.json] [--db ./data/media.db]++ *++ * Default: read-only — checks local files and database consistency, writes nothing.++ * --verify also HEADs every B2/R2 location with the storage engine's S3 client (MEDIA_B2_* / MEDIA_R2_*)++ * and records the result on media_locations (state, size, verified_at).++ * --hash sha256 local files of objects that carry a content hash (≤ 512 MB each).++ * Exit code 0 when clean, 1 when issues were found.++ */++const fs = require('fs');++const path = require('path');++++const args = process.argv.slice(2);++const has = (name) => args.includes(`--${name}`);++const arg = (name, dflt) => { const i = args.indexOf(`--${name}`); return i >= 0 ? args[i + 1] : dflt; };++if (arg('db')) process.env.DB_PATH = path.resolve(arg('db'));++++const { reconcile, summarize } = require('../server/objects/reconcile');++++(async () => {++ if (has('verify')) await require('../server/vod/vod-storage').checkProviders().catch(() => {});++ const report = await reconcile({ verify: has('verify'), hash: has('hash'), appId: arg('app', null) });++ if (arg('out')) fs.writeFileSync(arg('out'), JSON.stringify(report, null, 2));++ console.log(has('json') ? JSON.stringify(report, null, 2) : summarize(report));++ require('../server/db/database').close();++ process.exit(report.total_issues ? 1 : 0);++})().catch((err) => { console.error(err); process.exit(2); });@@@ -214,6 +214,7 @@ router.delete('/vods/bulk', async (req, res) => {try {const vod = Number.isFinite(id) ? db.getVodById(id, req.appId) : null;if (!vod) { result.error = 'VOD not found'; results.push(result); continue; }++ if (require('../objects/model').isHeldRow(vod)) { result.error = 'VOD is under a retention hold'; results.push(result); continue; }if (recorder.isRecording(vod.id)) recorder.stopRecording(vod.id);try { require('../vod/routes').activeChunkUploads.delete(vod.id); } catch { /* */ }@@@ -118,10 +118,11 @@ function createIngestHandler({ db, config, screenshotsDir, generateSlug, log = cconst file = path.join(screenshotsDir, `avatar-n${parseInt(user_id, 10) || 0}-${Date.now()}-${crypto.randomBytes(4).toString('hex')}.webp`);fs.writeFileSync(file, pic.buffer);const slug = generateSlug();-- db.run(`INSERT INTO pastes (app_id, slug, user_id, type, title, content, language, visibility, screenshot_path, metadata, ip_address)++ const info = db.run(`INSERT INTO pastes (app_id, slug, user_id, type, title, content, language, visibility, screenshot_path, metadata, ip_address)VALUES ('network', ?, ?, 'screenshot', ?, '', 'text', 'unlisted', ?, ?, NULL)`,[slug, parseInt(user_id, 10) || null, `Avatar${username ? ' of ' + String(username).slice(0, 40) : ''}`, file,JSON.stringify({ kind: 'avatar', source_host: (() => { try { return new URL(url).hostname; } catch { return null; } })(), size_bytes: pic.buffer.length, mime_type: 'image/webp' })]);++ if (info && typeof db.syncObject === 'function') db.syncObject('paste', info.lastInsertRowid); // the avatar's media_objectres.json({ ok: true, slug, url: `${config.publicUrl}/p/${slug}/screenshot`, width: pic.width, height: pic.height, bytes: pic.buffer.length });} catch (err) {log.warn('[Avatar ingest]', err.message);@@@ -43,6 +43,20 @@ const config = {maxSizeMb: intEnv('MAX_FILE_SIZE_MB', 256),},++ // Canonical object model (Wave 4): native v2 uploads, signed delivery, retention,++ // and the public playback object-size invariant (policy, not a copied constant).++ objects: {++ path: process.env.OBJECTS_PATH || './data/objects',++ maxUploadMb: intEnv('MEDIA_OBJECT_MAX_MB', 256), // single-part upload limit++ retentionDays: intEnv('MEDIA_DELETE_RETENTION_DAYS', 30), // soft-deleted bytes kept this long++ signingSecret: process.env.MEDIA_SIGNING_SECRET || '',++ signedUrlTtlS: intEnv('MEDIA_SIGNED_URL_TTL_S', 300),++ uploadTokenTtlS: intEnv('MEDIA_UPLOAD_TOKEN_TTL_S', 3600),++ publicMaxMb: intEnv('MEDIA_PUBLIC_OBJECT_MAX_MB', 500),++ publicTargetMb: intEnv('MEDIA_PUBLIC_OBJECT_TARGET_MB', 256),++ publicWarnMb: intEnv('MEDIA_PUBLIC_OBJECT_WARN_MB', 384),++ },++rtp: {portMin: intEnv('RTP_PORT_MIN', 12000),portMax: intEnv('RTP_PORT_MAX', 12199),@@@ -71,6 +85,7 @@ config.pastes.path = path.resolve(config.pastes.path);config.assets.path = path.resolve(config.assets.path);config.thumbnails.path = path.resolve(config.thumbnails.path);config.files.path = path.resolve(config.files.path);++config.objects.path = path.resolve(config.objects.path);config.db.path = path.resolve(config.db.path);module.exports = config;@@@ -28,6 +28,7 @@ function getDb() {const schema = fs.readFileSync(path.join(__dirname, 'schema.sql'), 'utf8');database.exec(schema);migrateColumns();++ ensureObjectTriggers();normalizeThumbnailUrls();seedSettings();recoverInterruptedClips();@@@ -71,8 +72,10 @@ function normalizeThumbnailUrls() {// (CREATE TABLE IF NOT EXISTS won't add columns to an existing table).function migrateColumns() {const wanted = {-- vods: [['managed_stream_id', 'INTEGER']],-- clips: [['channel_user_id', 'INTEGER'], ['cut_error', 'TEXT'], ['cut_attempts', 'INTEGER DEFAULT 0'], ['cut_next_at', 'DATETIME']],++ vods: [['managed_stream_id', 'INTEGER'], ['object_id', 'TEXT']],++ clips: [['channel_user_id', 'INTEGER'], ['cut_error', 'TEXT'], ['cut_attempts', 'INTEGER DEFAULT 0'], ['cut_next_at', 'DATETIME'], ['object_id', 'TEXT']],++ files: [['object_id', 'TEXT']],++ pastes: [['object_id', 'TEXT']],};for (const [table, cols] of Object.entries(wanted)) {const existing = database.prepare(`PRAGMA table_info(${table})`).all().map(c => c.name);@@@ -84,6 +87,43 @@ function migrateColumns() {}}++/**++ * Object-model guards that must hold on EVERY delete path, including the many++ * inherited ones that DELETE a vods/clips/files/pastes row directly:++ * - a row whose object is under an unreleased retention hold cannot be deleted;++ * - deleting a projected row marks its media_object deleted (bytes accounting++ * and reconciliation keep working without touching each call site).++ * Created here, after migrateColumns, because they reference object_id.++ */++function ensureObjectTriggers() {++ const held = (ref) => `EXISTS (SELECT 1 FROM media_holds WHERE object_id = ${ref} AND released_at IS NULL)`;++ for (const table of ['vods', 'clips', 'files', 'pastes']) {++ database.exec(`++ CREATE TRIGGER IF NOT EXISTS trg_${table}_hold_guard BEFORE DELETE ON ${table}++ WHEN OLD.object_id IS NOT NULL AND ${held('OLD.object_id')}++ BEGIN SELECT RAISE(ABORT, 'media object is under a retention hold'); END;++ CREATE TRIGGER IF NOT EXISTS trg_${table}_object_deleted AFTER DELETE ON ${table}++ WHEN OLD.object_id IS NOT NULL++ BEGIN++ UPDATE media_objects SET lifecycle_status = 'deleted', deleted_at = COALESCE(deleted_at, CURRENT_TIMESTAMP),++ updated_at = CURRENT_TIMESTAMP++ WHERE id = OLD.object_id AND lifecycle_status != 'deleted';++ END;`);++ }++ database.exec(`++ CREATE TRIGGER IF NOT EXISTS trg_media_objects_hold_guard BEFORE UPDATE OF lifecycle_status ON media_objects++ WHEN NEW.lifecycle_status = 'deleted' AND OLD.lifecycle_status != 'deleted' AND ${held('OLD.id')}++ BEGIN SELECT RAISE(ABORT, 'media object is under a retention hold'); END;++ CREATE TRIGGER IF NOT EXISTS trg_media_objects_hold_delete BEFORE DELETE ON media_objects++ WHEN ${held('OLD.id')}++ BEGIN SELECT RAISE(ABORT, 'media object is under a retention hold'); END;`);++}++++/** Keep a projected row's media_object current. Never throws — the legacy write already happened. */++function _syncObject(kind, id) {++ return require('../objects/model').safeSync(kind, id);++}++function seedSettings() {const defaults = [// Clip system@@@ -317,7 +357,9 @@ function _normVisibility(v) { return VALID_VISIBILITY.has(v) ? v : 'public'; }// Set VOD/clip visibility; is_public mirrors (1 iff public) so listing filters hold.function setVodVisibility(vodId, visibility) {const vis = _normVisibility(visibility);-- return run('UPDATE vods SET visibility = ?, is_public = ? WHERE id = ?', [vis, vis === 'public' ? 1 : 0, vodId]);++ const r = run('UPDATE vods SET visibility = ?, is_public = ? WHERE id = ?', [vis, vis === 'public' ? 1 : 0, vodId]);++ _syncObject('vod', vodId);++ return r;}function updateVodHealth(vodId, { status, score, issues = [], probeDuration, probeFormat, quarantine = false, keepPublic = false }) {@@@ -335,7 +377,9 @@ function updateVodHealth(vodId, { status, score, issues = [], probeDuration, proupdates.push("last_health_scan_at = datetime('now')");params.push(vodId);if (!updates.length) return null;-- return run(`UPDATE vods SET ${updates.join(', ')} WHERE id = ?`, params);++ const r = run(`UPDATE vods SET ${updates.join(', ')} WHERE id = ?`, params);++ _syncObject('vod', vodId);++ return r;}function repairVodDuration(vodId, duration, fileSize) {@@@ -380,13 +424,15 @@ function vodStatus(vod) {function createClip({ app_id, vod_id, stream_id, user_id, channel_user_id, title, description, file_path, thumbnail_url, start_time, end_time, duration_seconds, is_public, auto_generated, status }) {const pub = (is_public === 0 || is_public === false) ? 0 : 1;-- return run(++ const r = run(`INSERT INTO clips (app_id, vod_id, stream_id, user_id, channel_user_id, title, description, file_path, thumbnail_url, start_time, end_time, duration_seconds, is_public, visibility, auto_generated, status)VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,[app_id, vod_id || null, stream_id || null, user_id || null, channel_user_id || null, title || 'Untitled Clip', description || '',file_path || '', thumbnail_url || null, start_time || 0, end_time || 0, duration_seconds || 0,pub, pub ? 'public' : 'unlisted', auto_generated ? 1 : 0, status || 'ready']);++ _syncObject('clip', r.lastInsertRowid);++ return r;}function getClipById(id, appId = null) {@@@ -423,7 +469,9 @@ function countClips(appId, filters = {}) {function setClipVisibility(clipId, visibility) {const vis = _normVisibility(visibility);-- return run('UPDATE clips SET visibility = ?, is_public = ? WHERE id = ?', [vis, vis === 'public' ? 1 : 0, clipId]);++ const r = run('UPDATE clips SET visibility = ?, is_public = ? WHERE id = ?', [vis, vis === 'public' ? 1 : 0, clipId]);++ _syncObject('clip', clipId);++ return r;}function findDuplicateClip({ appId, streamId = null, vodId = null, startTime = 0, endTime = 0, startWindow = 8, endWindow = 10, createdSinceMinutes = 10 }) {@@@ -564,11 +612,13 @@ function getRecentPasteCommentsByIp(ip, seconds = 10) {// ── File helpers ─────────────────────────────────────────────function createFile({ key, app_id, user_id, original_name, size, mime, sha256 }) {-- return run(++ const r = run(`INSERT INTO files (key, app_id, user_id, original_name, size, mime, sha256)VALUES (?, ?, ?, ?, ?, ?, ?)`,[key, app_id, user_id || null, original_name || null, size || 0, mime || 'application/octet-stream', sha256 || null]);++ _syncObject('file', key);++ return r;}function getFileByKey(key, appId = null) {@@@ -651,7 +701,7 @@ function importLegacyRows(table, rows, appId = 'live') {}module.exports = {-- getDb, run, get, all, close,++ getDb, run, get, all, close, syncObject: _syncObject,getSetting, setSetting,// appshashApiKey, getApp, listApps, upsertApp, appAllowedOrigins,@@@ -53,6 +53,7 @@ CREATE TABLE IF NOT EXISTS vods (ai_transcript TEXT,ai_analyzed_at DATETIME,meta_json TEXT DEFAULT '{}',++ object_id TEXT, -- media_objects.id (Wave 4 object model)created_at DATETIME DEFAULT CURRENT_TIMESTAMP);CREATE INDEX IF NOT EXISTS idx_vods_app ON vods(app_id, created_at DESC);@@@ -84,6 +85,7 @@ CREATE TABLE IF NOT EXISTS clips (ai_overview TEXT,ai_transcript TEXT,ai_analyzed_at DATETIME,++ object_id TEXT, -- media_objects.idcreated_at DATETIME DEFAULT CURRENT_TIMESTAMP);CREATE INDEX IF NOT EXISTS idx_clips_app ON clips(app_id, created_at DESC);@@@ -126,6 +128,7 @@ CREATE TABLE IF NOT EXISTS pastes (ai_summary TEXT, -- kept for import; Live owns AI generationai_tags TEXT,ai_analyzed_at DATETIME,++ object_id TEXT, -- media_objects.id of the screenshot/avatar bytes (text pastes have none)created_at DATETIME DEFAULT CURRENT_TIMESTAMP,updated_at DATETIME DEFAULT CURRENT_TIMESTAMP,FOREIGN KEY (forked_from) REFERENCES pastes(id) ON DELETE SET NULL@@@ -169,6 +172,7 @@ CREATE TABLE IF NOT EXISTS files (size INTEGER NOT NULL DEFAULT 0,mime TEXT DEFAULT 'application/octet-stream',sha256 TEXT,++ object_id TEXT, -- media_objects.idcreated_at DATETIME DEFAULT CURRENT_TIMESTAMP);CREATE INDEX IF NOT EXISTS idx_files_app ON files(app_id, created_at DESC);@@@ -202,3 +206,110 @@ CREATE TABLE IF NOT EXISTS assets ();CREATE INDEX IF NOT EXISTS idx_assets_app_kind ON assets(app_id, kind, created_at DESC);CREATE UNIQUE INDEX IF NOT EXISTS idx_assets_identity ON assets(app_id, kind, name, channel_username);++++-- ── Canonical object model (roadmap Wave 4; docs/object-model.md) ──++-- Every stored byte-blob is a media_object. The vods/clips/files/pastes rows++-- above are typed projections over it (their object_id column), so the old++-- APIs and URLs keep working while new code speaks in med_<ULID> ids.++CREATE TABLE IF NOT EXISTS media_objects (++ id TEXT PRIMARY KEY, -- med_<ULID> (time-sortable)++ app_id TEXT NOT NULL, -- tenant++ namespace TEXT NOT NULL, -- capability namespace (= app_id today; projects later)++ kind TEXT NOT NULL CHECK(kind IN ('vod', 'clip', 'file', 'thumbnail', 'screenshot', 'avatar', 'asset')),++ owner_subject TEXT, -- canonical subject (usr_<ULID>) when known++ owner_app TEXT, -- legacy owner: the app whose user-id space owner_user_id is in++ owner_user_id INTEGER,++ visibility TEXT NOT NULL DEFAULT 'private' CHECK(visibility IN ('public', 'unlisted', 'private')),++ lifecycle_status TEXT NOT NULL DEFAULT 'uploading' CHECK(lifecycle_status IN ('uploading', 'ready', 'failed', 'archived', 'deleted')),++ mime_type TEXT,++ size_bytes INTEGER NOT NULL DEFAULT 0,++ content_hash TEXT, -- sha256 hex++ canonical_provider TEXT, -- local | b2 | r2++ canonical_key TEXT,++ legacy_ref TEXT UNIQUE, -- legacy:<app>:<kind>:<id> for projected rows; NULL = native v2 object++ metadata TEXT NOT NULL DEFAULT '{}',++ created_at DATETIME DEFAULT CURRENT_TIMESTAMP,++ updated_at DATETIME DEFAULT CURRENT_TIMESTAMP,++ deleted_at DATETIME++);++CREATE INDEX IF NOT EXISTS idx_media_objects_app ON media_objects(app_id, id);++CREATE INDEX IF NOT EXISTS idx_media_objects_kind ON media_objects(app_id, kind);++CREATE INDEX IF NOT EXISTS idx_media_objects_owner ON media_objects(owner_subject);++CREATE INDEX IF NOT EXISTS idx_media_objects_lifecycle ON media_objects(lifecycle_status);++++-- Where the bytes are. One row per provider copy; the canonical one is named on the object.++CREATE TABLE IF NOT EXISTS media_locations (++ id INTEGER PRIMARY KEY AUTOINCREMENT,++ object_id TEXT NOT NULL,++ provider TEXT NOT NULL CHECK(provider IN ('local', 'b2', 'r2')),++ bucket TEXT,++ key TEXT NOT NULL, -- absolute path for local, object key for b2/r2++ storage_class TEXT, -- hot (local) | cold (b2 canonical) | cache (r2)++ state TEXT NOT NULL DEFAULT 'pending' CHECK(state IN ('present', 'missing', 'pending', 'corrupt')),++ checksum TEXT,++ size_bytes INTEGER,++ verified_at DATETIME,++ created_at DATETIME DEFAULT CURRENT_TIMESTAMP,++ updated_at DATETIME DEFAULT CURRENT_TIMESTAMP,++ UNIQUE(object_id, provider)++);++++CREATE TABLE IF NOT EXISTS media_relationships (++ id INTEGER PRIMARY KEY AUTOINCREMENT,++ from_object_id TEXT NOT NULL,++ relation TEXT NOT NULL, -- clip_of | thumbnail_of | derived_from | screenshot_of++ to_object_id TEXT NOT NULL,++ metadata TEXT NOT NULL DEFAULT '{}',++ created_at DATETIME DEFAULT CURRENT_TIMESTAMP,++ UNIQUE(from_object_id, relation, to_object_id)++);++CREATE INDEX IF NOT EXISTS idx_media_relationships_to ON media_relationships(to_object_id, relation);++++CREATE TABLE IF NOT EXISTS media_variants (++ id INTEGER PRIMARY KEY AUTOINCREMENT,++ object_id TEXT NOT NULL, -- the source object++ variant_name TEXT NOT NULL, -- thumbnail | 720p | waveform | …++ derived_object_id TEXT NOT NULL,++ recipe TEXT, -- how it was made (name@version)++ created_at DATETIME DEFAULT CURRENT_TIMESTAMP,++ UNIQUE(object_id, variant_name)++);++++-- Generic derivative/maintenance jobs (schema only in this pass — no worker consumes it yet).++CREATE TABLE IF NOT EXISTS media_jobs (++ id INTEGER PRIMARY KEY AUTOINCREMENT,++ object_id TEXT,++ job_type TEXT NOT NULL,++ status TEXT NOT NULL DEFAULT 'queued' CHECK(status IN ('queued', 'running', 'done', 'failed', 'cancelled')),++ attempts INTEGER NOT NULL DEFAULT 0,++ checkpoint TEXT,++ error TEXT,++ created_at DATETIME DEFAULT CURRENT_TIMESTAMP,++ updated_at DATETIME DEFAULT CURRENT_TIMESTAMP++);++CREATE INDEX IF NOT EXISTS idx_media_jobs_status ON media_jobs(status, job_type);++++-- Retention holds: an object with an unreleased hold cannot be deleted or moved between tiers.++CREATE TABLE IF NOT EXISTS media_holds (++ id INTEGER PRIMARY KEY AUTOINCREMENT,++ object_id TEXT NOT NULL,++ kind TEXT NOT NULL CHECK(kind IN ('moderation', 'dmca', 'creator_pin', 'admin', 'evidence')),++ reason TEXT NOT NULL DEFAULT '',++ created_by TEXT,++ created_at DATETIME DEFAULT CURRENT_TIMESTAMP,++ released_at DATETIME,++ released_by TEXT++);++CREATE INDEX IF NOT EXISTS idx_media_holds_object ON media_holds(object_id, released_at);++++-- Public playback objects above the size policy (MEDIA_PUBLIC_OBJECT_*_MB). One row per object.++CREATE TABLE IF NOT EXISTS media_invariant_violations (++ id INTEGER PRIMARY KEY AUTOINCREMENT,++ object_id TEXT NOT NULL UNIQUE,++ level TEXT NOT NULL CHECK(level IN ('warn', 'violation')),++ size_bytes INTEGER NOT NULL,++ threshold_bytes INTEGER NOT NULL,++ detected_at DATETIME DEFAULT CURRENT_TIMESTAMP,++ last_seen_at DATETIME DEFAULT CURRENT_TIMESTAMP,++ resolved_at DATETIME++);@@@ -171,6 +171,7 @@ router.delete('/:key', tenantAuth({ allowUser: true }), (req, res) => {if (req.authType === 'user' && !(req.userId != null && row.user_id === req.userId)) {return res.status(403).json({ error: 'Not authorized to delete this file' });}++ if (require('../objects/model').isHeldRow(row)) return res.status(409).json({ error: 'File is under a retention hold', code: 'media.object.held' });const filePath = filePathForKey(row);try { if (fs.existsSync(filePath)) fs.unlinkSync(filePath); } catch { /* */ }@@@ -16,7 +16,7 @@ const auth = require('./auth');// ── Bootstrap ────────────────────────────────────────────────--for (const dir of [config.vod.path, config.vod.clipsPath, config.pastes.path, config.thumbnails.path, config.files.path]) {++for (const dir of [config.vod.path, config.vod.clipsPath, config.pastes.path, config.thumbnails.path, config.files.path, config.objects.path]) {if (!fs.existsSync(dir)) fs.mkdirSync(dir, { recursive: true });}@@@ -27,6 +27,8 @@ auth.startJwksRefresh();const app = express();app.set('trust proxy', true);++// Object API v2 content uploads read the raw request body, so they sit ahead of the body parsers.++app.put('/api/v2/:app/objects/:id/content', ...require('./objects/routes').contentHandlers);app.use(express.json({ limit: '2mb' }));// ── Visitor sign-in (OAuth client `media` on the Network; same module as Community/Tools) ──@@@ -66,7 +68,7 @@ app.use('/auth', userAuth.createAuthRoutes(userAuthConfig, userAuth.createAuthClapp.use(express.urlencoded({ extended: true, limit: '2mb' }));// CORS preflight for tenant API routes (per-app allow-list; auth-less OPTIONS).--app.options('/api/v1/:app/*', (req, res) => {++app.options(['/api/v1/:app/*', '/api/v2/:app/*'], (req, res) => {const origin = req.headers.origin;const appRow = db.getApp(String(req.params.app || ''));if (origin && appRow && db.appAllowedOrigins(appRow).includes(origin)) {@@@ -122,6 +124,8 @@ app.use('/api/v1/:app/files', require('./files/routes'));app.use('/api/v1/:app/thumbnails', require('./thumbnails/routes'));app.use('/api/v1/:app/assets', require('./assets/routes'));app.use('/api/v1/:app/admin/storage', require('./admin/routes'));++app.use('/api/v2/:app/objects', require('./objects/routes')); // canonical object API (docs/object-model.md)++app.use('/o', require('./objects/routes').publicRouter); // object bytes (public, or signed)app.use('/', require('./public/routes')); // /v /c /p /t /f// eslint-disable-next-line no-unused-vars@@@ -159,6 +163,18 @@ every(5 * 60 * 1000, () => {});every(60 * 1000, () => recorder.checkDisk()); // disk guardianevery(60 * 60 * 1000, () => thumbService.cleanupOldThumbnails()); // stale live thumbs++// Object model: purge native objects whose soft-delete retention has passed (held ones are kept).++every(60 * 60 * 1000, () => {++ try { const n = require('./objects/model').purgeExpired(); if (n) console.log(`[Objects] Purged ${n} expired deleted object(s)`); } catch (err) { console.warn('[Objects] purge:', err.message); }++});++// Project rows that have no media_object yet (first boot after the upgrade: all of them).++setTimeout(() => {++ try {++ const bf = require('./objects/backfill');++ const r = bf.backfill({ onlyMissing: true });++ if (r.totals.created || r.totals.updated || r.errors.length) console.log(`[Objects] Backfill: ${bf.summarize(r)}`);++ } catch (err) { console.warn('[Objects] Backfill failed:', err.message); }++}, 15 * 1000).unref?.();// Recover from an unclean shutdown: rows stuck in is_recording with no live// ffmpeg are finalized from whatever hit the disk.nnew file mode 100644@@@ -0,0 +1,110 @@++/**++ * OpenVibe.Media — object backfill++ *++ * One media_object (+ locations, relationships, thumbnail variant) per existing++ * vod, clip, file, screenshot/avatar paste and vod/clip thumbnail, derived by the++ * same sync functions the write hooks use. Idempotent: objects are keyed by their++ * legacy ref, so a re-run updates in place and creates nothing new.++ *++ * Never touches bytes and never calls B2/R2: local copies are 'present' when the++ * file exists on disk and 'missing' when it does not; remote copies stay 'pending'++ * until `scripts/reconcile-objects.js --verify` HEADs them.++ *++ * Runs in one transaction; dryRun rolls it back and only reports.++ */++'use strict';++++const db = require('../db/database');++const model = require('./model');++++const SETTING_KEY = 'objects.backfill.last_report';++++function emptyCounts() {++ return { seen: 0, created: 0, updated: 0, skipped: 0 };++}++++function backfill({ dryRun = false, onlyMissing = false } = {}) {++ const report = {++ started_at: new Date().toISOString(), dry_run: !!dryRun, only_missing: !!onlyMissing,++ counts: { vod: emptyCounts(), clip: emptyCounts(), file: emptyCounts(), screenshot: emptyCounts(), avatar: emptyCounts(), thumbnail: emptyCounts() },++ locations: { present: 0, missing: 0, pending: 0 },++ skipped: [],++ errors: [],++ };++ const where = onlyMissing ? ' WHERE object_id IS NULL' : '';++++ const tally = (kind, table, id, r) => {++ const c = report.counts[kind];++ c.seen++;++ if (!r) return;++ if (r.skipped) { c.skipped++; report.skipped.push({ table, id, reason: r.skipped }); return; }++ c[r.created ? 'created' : 'updated']++;++ for (const s of r.locations || []) if (s in report.locations) report.locations[s]++;++ };++ const thumb = (table, id, r) => {++ if (!r || !r.thumbnail) return;++ tally('thumbnail', `${table}.thumbnail_url`, id, r.thumbnail);++ };++ const d = db.getDb();++ // One savepoint per row: a row that fails leaves nothing half-written behind.++ const each = (table, sql, fn) => {++ const one = d.transaction(fn);++ for (const row of db.all(sql)) {++ try { one(row); } catch (err) { report.errors.push({ table, id: row.id ?? row.key, error: err.message }); }++ }++ };++++ d.exec('BEGIN');++ try {++ // VODs before clips, so clip_of can point at the vod's object.++ each('vods', `SELECT * FROM vods${where} ORDER BY id`, (row) => {++ const r = model.syncVod(row);++ tally('vod', 'vods', row.id, r);++ thumb('vods', row.id, r);++ });++ each('clips', `SELECT * FROM clips${where} ORDER BY id`, (row) => {++ const r = model.syncClip(row);++ tally('clip', 'clips', row.id, r);++ thumb('clips', row.id, r);++ });++ each('files', `SELECT * FROM files${where} ORDER BY created_at, key`, (row) => {++ tally('file', 'files', row.key, model.syncFile(row));++ });++ each('pastes', `SELECT * FROM pastes${where ? where + ' AND' : ' WHERE'} type = 'screenshot' ORDER BY id`, (row) => {++ const r = model.syncPaste(row);++ const kind = (r && r.kind) || (String(row.metadata || '').includes('"kind":"avatar"') ? 'avatar' : 'screenshot');++ tally(kind, 'pastes', row.id, r);++ });++ if (dryRun) d.exec('ROLLBACK'); else d.exec('COMMIT');++ } catch (err) {++ try { d.exec('ROLLBACK'); } catch { /* already rolled back */ }++ throw err;++ }++++ report.finished_at = new Date().toISOString();++ report.totals = Object.values(report.counts).reduce((t, c) => {++ for (const k of Object.keys(t)) t[k] += c[k];++ return t;++ }, emptyCounts());++ if (!dryRun && (!onlyMissing || report.totals.created || report.totals.updated)) {++ // Keep the latest report (skipped rows capped) for operators: media_settings.objects.backfill.last_report.++ const stored = { ...report, skipped: report.skipped.slice(0, 500), skipped_truncated: report.skipped.length > 500 };++ db.run(`INSERT INTO media_settings (key, value, description, type) VALUES (?, ?, 'Last object backfill report', 'json')++ ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = CURRENT_TIMESTAMP`, [SETTING_KEY, JSON.stringify(stored)]);++ }++ return report;++}++++function lastReport() {++ const row = db.get('SELECT value FROM media_settings WHERE key = ?', [SETTING_KEY]);++ return row ? model.parseJson(row.value, null) : null;++}++++function summarize(report) {++ const parts = Object.entries(report.counts).filter(([, c]) => c.seen)++ .map(([k, c]) => `${k} ${c.created}+/${c.updated}~/${c.skipped} skipped`);++ return `${report.dry_run ? '[dry run] ' : ''}${parts.join(', ') || 'nothing to do'}; locations present ${report.locations.present}, `++ + `missing ${report.locations.missing}, pending ${report.locations.pending}${report.errors.length ? `; ${report.errors.length} error(s)` : ''}`;++}++++module.exports = { backfill, lastReport, summarize, SETTING_KEY };nnew file mode 100644@@@ -0,0 +1,94 @@++/**++ * OpenVibe.Media — public object-size invariant (roadmap W4 deliverable 5)++ *++ * No public playback object above MEDIA_PUBLIC_OBJECT_MAX_MB (default 500),++ * target MEDIA_PUBLIC_OBJECT_TARGET_MB (256), warn at MEDIA_PUBLIC_OBJECT_WARN_MB++ * (384). Policy, not a constant: the thresholds are configuration, the++ * validator is pure, and violations are recorded in media_invariant_violations++ * (one row per object, resolved when it drops back under). This pass reports++ * and refuses new oversized public uploads; it never re-encodes.++ */++'use strict';++++const db = require('../db/database');++const config = require('../config');++++const MB = 1024 * 1024;++const PLAYBACK_KINDS = ['vod', 'clip'];++++function thresholds(cfg = config.objects) {++ return { targetBytes: cfg.publicTargetMb * MB, warnBytes: cfg.publicWarnMb * MB, maxBytes: cfg.publicMaxMb * MB };++}++++/** ok ≤ target < above_target ≤ warn < warn-level ≤ max < violation */++function classify(sizeBytes, t = thresholds()) {++ const n = Number(sizeBytes) || 0;++ if (n > t.maxBytes) return 'violation';++ if (n > t.warnBytes) return 'warn';++ if (n > t.targetBytes) return 'above_target';++ return 'ok';++}++++/** Playback bytes anyone with the link can fetch: ready vods/clips that are public or unlisted. */++function isPublicPlayback(obj) {++ return !!obj && PLAYBACK_KINDS.includes(obj.kind) && obj.visibility !== 'private' && obj.lifecycle_status === 'ready';++}++++/** Would an object of this kind/visibility/size break the invariant (a public playback object above max)? */++function wouldViolate({ kind, visibility, size_bytes }, t = thresholds()) {++ return PLAYBACK_KINDS.includes(kind) && visibility !== 'private' && classify(size_bytes, t) === 'violation';++}++++function validate(obj, t = thresholds()) {++ const level = classify(obj.size_bytes, t);++ const threshold = level === 'violation' ? t.maxBytes : level === 'warn' ? t.warnBytes : level === 'above_target' ? t.targetBytes : null;++ return { object_id: obj.id, kind: obj.kind, visibility: obj.visibility, size_bytes: Number(obj.size_bytes) || 0,++ level, threshold_bytes: threshold, public_playback: isPublicPlayback(obj) };++}++++/** Upsert (warn/violation) or resolve this object's violation row. Returns the level recorded, or null. */++function record(obj, t = thresholds()) {++ if (!obj) return null;++ const v = validate(obj, t);++ const open = db.get('SELECT * FROM media_invariant_violations WHERE object_id = ?', [obj.id]);++ if (v.public_playback && (v.level === 'warn' || v.level === 'violation')) {++ if (open) {++ db.run(`UPDATE media_invariant_violations SET level = ?, size_bytes = ?, threshold_bytes = ?, last_seen_at = CURRENT_TIMESTAMP,++ resolved_at = NULL, detected_at = CASE WHEN resolved_at IS NULL THEN detected_at ELSE CURRENT_TIMESTAMP END++ WHERE id = ?`, [v.level, v.size_bytes, v.threshold_bytes, open.id]);++ } else {++ db.run('INSERT INTO media_invariant_violations (object_id, level, size_bytes, threshold_bytes) VALUES (?, ?, ?, ?)',++ [obj.id, v.level, v.size_bytes, v.threshold_bytes]);++ }++ return v.level;++ }++ if (open && !open.resolved_at) db.run('UPDATE media_invariant_violations SET resolved_at = CURRENT_TIMESTAMP WHERE id = ?', [open.id]);++ return null;++}++++/**++ * Scan every public playback object. Returns { thresholds, counts, objects } where++ * objects lists everything above target (largest first). record=false is a dry run.++ */++function scan({ record: write = true, appId = null } = {}) {++ const t = thresholds();++ const rows = db.all(`SELECT * FROM media_objects WHERE kind IN ('vod', 'clip') AND visibility != 'private' AND lifecycle_status = 'ready'++ ${appId ? 'AND app_id = ?' : ''} ORDER BY size_bytes DESC`, appId ? [appId] : []);++ const counts = { ok: 0, above_target: 0, warn: 0, violation: 0 };++ const objects = [];++ for (const obj of rows) {++ const v = validate(obj, t);++ counts[v.level]++;++ if (v.level !== 'ok') objects.push({ ...v, app_id: obj.app_id, legacy_ref: obj.legacy_ref });++ if (write) record(obj, t);++ }++ // Objects that stopped being public playback (made private, deleted) resolve too.++ if (write) {++ const stale = db.all(`SELECT o.*, v.id AS violation_id FROM media_invariant_violations v LEFT JOIN media_objects o ON o.id = v.object_id++ WHERE v.resolved_at IS NULL`);++ for (const s of stale) if (!isPublicPlayback(s)) db.run('UPDATE media_invariant_violations SET resolved_at = CURRENT_TIMESTAMP WHERE id = ?', [s.violation_id]);++ }++ return { thresholds: t, counts, objects };++}++++module.exports = { thresholds, classify, isPublicPlayback, wouldViolate, validate, record, scan, MB };nnew file mode 100644@@@ -0,0 +1,550 @@++/**++ * OpenVibe.Media — canonical object model (roadmap Wave 4; docs/object-model.md)++ *++ * Every stored blob is a media_object (id med_<ULID>) with one media_locations++ * row per provider copy. The inherited vods/clips/files/pastes rows are typed++ * projections over objects: each carries object_id, and the sync*() functions++ * below derive the object + locations from the row as it is right now. The same++ * functions back the one-off backfill and the write hooks on the old APIs, so++ * the model cannot drift between "imported" and "written since".++ *++ * Location states: present (verified: the local file exists / a HEAD answered),++ * missing (verified absent), pending (believed there, not yet verified — every++ * remote copy until reconciliation --verify), corrupt (size/hash mismatch).++ */++'use strict';++++const fs = require('fs');++const path = require('path');++const { ids } = require('openvibe-contracts');++const db = require('../db/database');++const config = require('../config');++++const KINDS = ['vod', 'clip', 'file', 'thumbnail', 'screenshot', 'avatar', 'asset'];++const VISIBILITIES = ['public', 'unlisted', 'private'];++const LIFECYCLES = ['uploading', 'ready', 'failed', 'archived', 'deleted'];++const HOLD_KINDS = ['moderation', 'dmca', 'creator_pin', 'admin', 'evidence'];++const STORAGE_CLASS = { local: 'hot', b2: 'cold', r2: 'cache' };++++const MIME_BY_EXT = {++ '.webm': 'video/webm', '.mp4': 'video/mp4', '.mkv': 'video/x-matroska', '.mov': 'video/quicktime', '.ogg': 'video/ogg',++ '.jpg': 'image/jpeg', '.jpeg': 'image/jpeg', '.png': 'image/png', '.webp': 'image/webp', '.gif': 'image/gif',++ '.mp3': 'audio/mpeg', '.wav': 'audio/wav', '.txt': 'text/plain', '.json': 'application/json', '.pdf': 'application/pdf',++};++++class HeldError extends Error {++ constructor(objectId) { super('media object is under a retention hold'); this.code = 'media.object.held'; this.objectId = objectId; }++}++++// ── Small helpers ────────────────────────────────────────────++++function mimeFor(p, fallback = 'application/octet-stream') {++ return MIME_BY_EXT[path.extname(String(p || '')).toLowerCase()] || fallback;++}++++function parseJson(s, fallback) {++ if (s == null || s === '') return fallback;++ try { return JSON.parse(s); } catch { return fallback; }++}++++/** SQLite 'YYYY-MM-DD HH:MM:SS' (UTC) → epoch ms, so backfilled ids sort by original creation time. */++function toMs(createdAt) {++ const ms = Date.parse(String(createdAt || '').replace(' ', 'T') + (/[zZ]|[+-]\d\d:?\d\d$/.test(String(createdAt || '')) ? '' : 'Z'));++ return Number.isFinite(ms) ? ms : Date.now();++}++++function statFile(p) {++ if (!p) return null;++ try { const st = fs.statSync(p); return st.isFile() ? { size: st.size } : null; } catch { return null; }++}++++function rowVisibility(row) {++ const v = row.visibility || (row.is_public ? 'public' : 'private');++ return VISIBILITIES.includes(v) ? v : 'private';++}++++function legacyRef(app, kind, id) {++ return ids.legacyMediaId(app, kind, id);++}++++function parseLegacyRef(ref) {++ const m = /^legacy:([^:]+):([^:]+):(.+)$/.exec(String(ref || ''));++ return m ? { app: m[1], kind: m[2], id: m[3] } : null;++}++++const vodStorage = () => require('../vod/vod-storage');++++// ── Reads ────────────────────────────────────────────────────++++function getObject(id) {++ return id ? db.get('SELECT * FROM media_objects WHERE id = ?', [id]) : null;++}++++function getObjectByLegacyRef(ref) {++ return ref ? db.get('SELECT * FROM media_objects WHERE legacy_ref = ?', [ref]) : null;++}++++/** An object by med_ id or legacy ref, scoped to one tenant (null when it belongs to another). */++function resolveObject(idOrRef, appId) {++ const key = String(idOrRef || '');++ const obj = key.startsWith('legacy:') ? getObjectByLegacyRef(key) : getObject(key);++ return obj && obj.app_id === appId ? obj : null;++}++++function listLocations(objectId) {++ return db.all('SELECT * FROM media_locations WHERE object_id = ? ORDER BY id', [objectId]);++}++++function listHolds(objectId, { includeReleased = false } = {}) {++ return db.all(`SELECT * FROM media_holds WHERE object_id = ?${includeReleased ? '' : ' AND released_at IS NULL'} ORDER BY id`, [objectId]);++}++++function isHeld(objectId) {++ if (!objectId) return false;++ return !!db.get('SELECT 1 AS x FROM media_holds WHERE object_id = ? AND released_at IS NULL LIMIT 1', [objectId]);++}++++/** Hold check for an inherited row (vods/clips/files/pastes) — false when it has no object yet. */++function isHeldRow(row) {++ try { return !!(row && row.object_id && isHeld(row.object_id)); } catch { return false; }++}++++// ── Writes ───────────────────────────────────────────────────++++const OBJECT_COLUMNS = ['app_id', 'namespace', 'kind', 'owner_subject', 'owner_app', 'owner_user_id', 'visibility',++ 'lifecycle_status', 'mime_type', 'size_bytes', 'content_hash', 'canonical_provider', 'canonical_key', 'legacy_ref', 'metadata', 'deleted_at'];++++function createObject(o) {++ if (!KINDS.includes(o.kind)) throw new Error(`unknown kind ${o.kind}`);++ const id = ids.newId('media', o.createdMs);++ db.run(`INSERT INTO media_objects (id, app_id, namespace, kind, owner_subject, owner_app, owner_user_id, visibility,++ lifecycle_status, mime_type, size_bytes, content_hash, canonical_provider, canonical_key, legacy_ref, metadata, created_at)++ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, COALESCE(?, CURRENT_TIMESTAMP))`,++ [id, o.app_id, o.namespace || o.app_id, o.kind, o.owner_subject || null, o.owner_app || null, o.owner_user_id ?? null,++ VISIBILITIES.includes(o.visibility) ? o.visibility : 'private', LIFECYCLES.includes(o.lifecycle_status) ? o.lifecycle_status : 'uploading',++ o.mime_type || null, Number(o.size_bytes) || 0, o.content_hash || null, o.canonical_provider || null, o.canonical_key || null,++ o.legacy_ref || null, JSON.stringify(o.metadata || {}), o.created_at || null]);++ return id;++}++++function updateObject(id, fields) {++ const sets = [], params = [];++ for (const [k, v] of Object.entries(fields)) {++ if (!OBJECT_COLUMNS.includes(k)) continue;++ sets.push(`${k} = ?`);++ params.push(k === 'metadata' && typeof v !== 'string' ? JSON.stringify(v || {}) : v);++ }++ if (!sets.length) return;++ params.push(id);++ db.run(`UPDATE media_objects SET ${sets.join(', ')}, updated_at = CURRENT_TIMESTAMP WHERE id = ?`, params);++}++++/**++ * Upsert one provider copy. A remote copy re-derived as 'pending' keeps what++ * reconciliation already verified about the same key (never downgrade knowledge).++ */++function upsertLocation(objectId, loc) {++ const existing = db.get('SELECT * FROM media_locations WHERE object_id = ? AND provider = ?', [objectId, loc.provider]);++ const verifiedAt = loc.verified ? new Date().toISOString().replace('T', ' ').slice(0, 19) : null;++ if (!existing) {++ db.run(`INSERT INTO media_locations (object_id, provider, bucket, key, storage_class, state, checksum, size_bytes, verified_at)++ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,++ [objectId, loc.provider, loc.bucket || null, loc.key, loc.storage_class || STORAGE_CLASS[loc.provider] || null,++ loc.state || 'pending', loc.checksum || null, loc.size_bytes ?? null, verifiedAt]);++ return;++ }++ const keep = existing.key === loc.key && (loc.state || 'pending') === 'pending' && existing.state !== 'pending';++ db.run(`UPDATE media_locations SET bucket = ?, key = ?, storage_class = ?, state = ?, checksum = ?, size_bytes = ?, verified_at = ?,++ updated_at = CURRENT_TIMESTAMP WHERE id = ?`,++ [loc.bucket ?? existing.bucket, loc.key, loc.storage_class || existing.storage_class,++ keep ? existing.state : (loc.state || 'pending'),++ loc.checksum ?? (existing.key === loc.key ? existing.checksum : null),++ keep ? existing.size_bytes : (loc.size_bytes ?? null),++ keep ? existing.verified_at : (verifiedAt || (existing.key === loc.key && loc.state === existing.state ? existing.verified_at : null)),++ existing.id]);++}++++function setLocationState(locationId, { state, size_bytes, verified = true }) {++ db.run(`UPDATE media_locations SET state = ?, size_bytes = COALESCE(?, size_bytes), verified_at = ${verified ? 'CURRENT_TIMESTAMP' : 'verified_at'},++ updated_at = CURRENT_TIMESTAMP WHERE id = ?`, [state, size_bytes ?? null, locationId]);++}++++function setRelationship(fromId, relation, toId, metadata = {}) {++ db.run(`INSERT INTO media_relationships (from_object_id, relation, to_object_id, metadata) VALUES (?, ?, ?, ?)++ ON CONFLICT(from_object_id, relation, to_object_id) DO UPDATE SET metadata = excluded.metadata`,++ [fromId, relation, toId, JSON.stringify(metadata)]);++}++++function setVariant(objectId, variantName, derivedId, recipe = null) {++ db.run(`INSERT INTO media_variants (object_id, variant_name, derived_object_id, recipe) VALUES (?, ?, ?, ?)++ ON CONFLICT(object_id, variant_name) DO UPDATE SET derived_object_id = excluded.derived_object_id, recipe = excluded.recipe`,++ [objectId, variantName, derivedId, recipe]);++}++++function getVariant(objectId, variantName) {++ return db.get('SELECT * FROM media_variants WHERE object_id = ? AND variant_name = ?', [objectId, variantName]);++}++++// ── Projection core ──────────────────────────────────────────++++/**++ * Create or update the object behind one inherited row and make its locations++ * exactly the derived set. Returns { id, created }.++ * p.locations [{ provider, bucket, key, state, size_bytes, checksum }]++ * p.canonical preferred canonical provider (falls back to the first location)++ */++function project(p) {++ const found = getObjectByLegacyRef(p.legacy_ref) || (p.existingId ? getObject(p.existingId) : null);++ const canon = p.locations.find(l => l.provider === p.canonical) || p.locations[0] || null;++ const fields = {++ app_id: p.app_id, namespace: p.app_id, kind: p.kind,++ owner_app: p.owner_app || p.app_id, owner_user_id: p.owner_user_id ?? null,++ visibility: p.visibility, lifecycle_status: p.lifecycle_status,++ mime_type: p.mime_type || null, size_bytes: Number(p.size_bytes) || 0, content_hash: p.content_hash || null,++ canonical_provider: canon ? canon.provider : null, canonical_key: canon ? canon.key : null,++ legacy_ref: p.legacy_ref, metadata: p.metadata || {},++ };++ let id, created = false;++ if (found) {++ id = found.id;++ // Knowledge the row doesn't carry survives a re-projection.++ if (!fields.content_hash) fields.content_hash = found.content_hash;++ fields.metadata = { ...parseJson(found.metadata, {}), ...fields.metadata };++ if (found.lifecycle_status === 'deleted') fields.deleted_at = null; // the row exists, so the object does++ updateObject(id, fields);++ } else {++ id = createObject({ ...fields, createdMs: toMs(p.created_at), created_at: p.created_at || null });++ created = true;++ }++ const keep = new Set(p.locations.map(l => l.provider));++ for (const loc of p.locations) upsertLocation(id, loc);++ for (const l of listLocations(id)) if (!keep.has(l.provider)) db.run('DELETE FROM media_locations WHERE id = ?', [l.id]);++ return { id, created };++}++++function localLocation(candidates, { mustExist = false } = {}) {++ for (const p of candidates.filter(Boolean)) {++ const st = statFile(p);++ if (st) return { provider: 'local', key: path.resolve(p), state: 'present', size_bytes: st.size, verified: true };++ }++ if (mustExist || !candidates[0]) return null;++ return { provider: 'local', key: path.resolve(candidates[0]), state: 'missing', size_bytes: null, verified: true };++}++++/** local + B2 canonical + R2 cache copies for a vods/clips row (same columns, same key scheme). */++function tieredLocations(row, localCandidates) {++ const vs = vodStorage();++ const provider = vs.providerOf(row);++ const remoteKnown = provider !== 'local' || !!row.storage_key;++ const locs = [];++ const local = row.file_path ? localLocation(localCandidates, { mustExist: provider !== 'local' }) : null;++ if (local) locs.push(local);++ if (remoteKnown && row.file_path) {++ const key = vs.keyForVod(row);++ locs.push({ provider: 'b2', bucket: vs.bucketFor('b2'), key, state: 'pending' });++ if (provider === 'r2') locs.push({ provider: 'r2', bucket: vs.bucketFor('r2'), key, state: 'pending' });++ }++ return { locs, canonical: remoteKnown ? 'b2' : 'local' };++}++++function linkRow(table, keyCol, keyVal, objectId) {++ db.run(`UPDATE ${table} SET object_id = ? WHERE ${keyCol} = ? AND (object_id IS NULL OR object_id != ?)`, [objectId, keyVal, objectId]);++}++++function recordInvariant(objectId) {++ try { require('./invariant').record(getObject(objectId)); } catch (err) { console.warn('[Objects] invariant record:', err.message); }++}++++// ── Per-kind projections ─────────────────────────────────────++++function vodLifecycle(row) {++ const s = db.vodStatus(row);++ if (s === 'ready' || s === 'failed') return s;++ return 'uploading';++}++++function syncVod(idOrRow) {++ const row = typeof idOrRow === 'object' ? idOrRow : db.get('SELECT * FROM vods WHERE id = ?', [idOrRow]);++ if (!row) return null;++ if (row.clips_only) return { skipped: 'clips-only recording (ephemeral, never published)' };++ const vs = vodStorage();++ const { locs, canonical } = tieredLocations(row, [row.file_path ? vs.localPathForVod(row) : null, row.file_path]);++ const local = locs.find(l => l.provider === 'local' && l.state === 'present');++ const r = project({++ legacy_ref: legacyRef(row.app_id, 'vod', row.id), existingId: row.object_id,++ app_id: row.app_id, kind: 'vod', owner_user_id: row.user_id,++ visibility: rowVisibility(row), lifecycle_status: vodLifecycle(row),++ mime_type: row.file_path ? mimeFor(row.file_path, 'video/webm') : null,++ size_bytes: Number(row.file_size) || (local ? local.size_bytes : 0),++ metadata: { title: row.title || null, duration_seconds: Number(row.duration_seconds) || 0, health_status: row.health_status || null,++ managed_stream_id: row.managed_stream_id || null, recording: !!row.is_recording },++ created_at: row.created_at, locations: locs, canonical,++ });++ linkRow('vods', 'id', row.id, r.id);++ recordInvariant(r.id);++ const thumbnail = syncThumbnail(row, r.id, 'vod');++ return { ...r, locations: locs.map(l => l.state), thumbnail };++}++++function syncClip(idOrRow) {++ const row = typeof idOrRow === 'object' ? idOrRow : db.get('SELECT * FROM clips WHERE id = ?', [idOrRow]);++ if (!row) return null;++ const clipsDir = path.resolve(config.vod.clipsPath);++ const { locs, canonical } = tieredLocations(row, [row.file_path || null, row.file_path ? path.join(clipsDir, path.basename(row.file_path)) : null]);++ const status = row.status || 'ready';++ const lifecycle = status === 'processing' ? 'uploading' : status === 'failed' ? 'failed' : (row.file_path ? 'ready' : 'failed');++ const local = locs.find(l => l.provider === 'local' && l.state === 'present');++ const r = project({++ legacy_ref: legacyRef(row.app_id, 'clip', row.id), existingId: row.object_id,++ app_id: row.app_id, kind: 'clip', owner_user_id: row.user_id,++ visibility: rowVisibility(row), lifecycle_status: lifecycle,++ mime_type: row.file_path ? mimeFor(row.file_path, 'video/webm') : null,++ size_bytes: local ? local.size_bytes : 0,++ metadata: { title: row.title || null, duration_seconds: Number(row.duration_seconds) || 0, start_time: row.start_time, end_time: row.end_time,++ channel_user_id: row.channel_user_id || null, auto_generated: !!row.auto_generated },++ created_at: row.created_at, locations: locs, canonical,++ });++ linkRow('clips', 'id', row.id, r.id);++ if (row.vod_id) {++ const vod = db.get('SELECT object_id FROM vods WHERE id = ?', [row.vod_id]);++ if (vod && vod.object_id) setRelationship(r.id, 'clip_of', vod.object_id, { start_time: row.start_time, end_time: row.end_time });++ }++ recordInvariant(r.id);++ const thumbnail = syncThumbnail(row, r.id, 'clip');++ return { ...r, locations: locs.map(l => l.state), thumbnail };++}++++function syncFile(keyOrRow) {++ const row = typeof keyOrRow === 'object' ? keyOrRow : db.get('SELECT * FROM files WHERE key = ?', [keyOrRow]);++ if (!row) return null;++ const local = localLocation([path.join(config.files.path, row.app_id, row.key)]);++ if (local.state === 'present' && row.sha256) local.checksum = row.sha256;++ const r = project({++ legacy_ref: legacyRef(row.app_id, 'file', row.key), existingId: row.object_id,++ app_id: row.app_id, kind: 'file', owner_user_id: row.user_id,++ visibility: 'public', // /f/:key serves every file without auth++ lifecycle_status: 'ready', mime_type: row.mime || 'application/octet-stream',++ size_bytes: Number(row.size) || 0, content_hash: row.sha256 || null,++ metadata: { filename: row.original_name || null },++ created_at: row.created_at, locations: [local], canonical: 'local',++ });++ linkRow('files', 'key', row.key, r.id);++ return { ...r, locations: [local.state] };++}++++/** Screenshot pastes (and avatars, which are stored as screenshot pastes) — the bytes only; text is Community's. */++function syncPaste(idOrRow) {++ const row = typeof idOrRow === 'object' ? idOrRow : db.get('SELECT * FROM pastes WHERE id = ?', [idOrRow]);++ if (!row) return null;++ if (row.type !== 'screenshot') return { skipped: 'text paste (no bytes)' };++ if (!row.screenshot_path) return { skipped: 'screenshot paste without a file path' };++ const meta = parseJson(row.metadata, {}) || {};++ const kind = meta.kind === 'avatar' ? 'avatar' : 'screenshot';++ const local = localLocation([row.screenshot_path]);++ const r = project({++ legacy_ref: legacyRef(row.app_id, kind === 'avatar' ? 'avatar' : 'paste', row.slug), existingId: row.object_id,++ app_id: row.app_id, kind, owner_user_id: row.user_id,++ visibility: VISIBILITIES.includes(row.visibility) ? row.visibility : 'public', lifecycle_status: 'ready',++ mime_type: meta.mime_type || mimeFor(row.screenshot_path, 'image/png'),++ size_bytes: local.state === 'present' ? local.size_bytes : (Number(meta.size_bytes) || 0),++ metadata: { title: row.title || null, slug: row.slug },++ created_at: row.created_at, locations: [local], canonical: 'local',++ });++ linkRow('pastes', 'id', row.id, r.id);++ return { ...r, kind, locations: [local.state] };++}++++/** Thumbnail file name behind a stored thumbnail_url, or null when it is not one of ours. */++function thumbFileFromUrl(u) {++ if (!u) return null;++ let p = String(u);++ if (/^https?:\/\//i.test(p)) {++ if (!p.startsWith(config.publicUrl + '/')) return null;++ p = p.slice(config.publicUrl.length);++ }++ const m = /^\/(?:t|api\/thumbnails)\/([A-Za-z0-9._-]+)$/.exec(p);++ return m ? m[1] : null;++}++++/**++ * A vod/clip's thumbnail is one object per parent, updated in place when the++ * picture is regenerated (live recordings refresh theirs every couple of minutes).++ */++function syncThumbnail(parentRow, parentObjectId, parentKind) {++ if (!parentRow.thumbnail_url) return null;++ const name = thumbFileFromUrl(parentRow.thumbnail_url);++ if (!name) return { skipped: 'external thumbnail url' };++ const local = localLocation([path.join(path.resolve(config.thumbnails.path), name)]);++ const variant = getVariant(parentObjectId, 'thumbnail');++ const parentVis = rowVisibility(parentRow);++ const r = project({++ legacy_ref: legacyRef(parentRow.app_id, 'thumbnail', name), existingId: variant ? variant.derived_object_id : null,++ app_id: parentRow.app_id, kind: 'thumbnail', owner_user_id: parentRow.user_id,++ // /t/<name> is served to anyone holding the name.++ visibility: parentVis === 'public' ? 'public' : 'unlisted', lifecycle_status: 'ready',++ mime_type: mimeFor(name, 'image/jpeg'), size_bytes: local.state === 'present' ? local.size_bytes : 0,++ metadata: { of: parentKind },++ created_at: parentRow.created_at, locations: [local], canonical: 'local',++ });++ setVariant(parentObjectId, 'thumbnail', r.id);++ setRelationship(r.id, 'thumbnail_of', parentObjectId);++ return { ...r, locations: [local.state] };++}++++function _syncTx(fn, arg) {++ return db.getDb().transaction(() => fn(arg))();++}++++/** Re-project one inherited row: kind vod|clip|file|paste, id = row id (file key for files). */++function sync(kind, id) {++ const fn = { vod: syncVod, clip: syncClip, file: syncFile, paste: syncPaste }[kind];++ if (!fn || id == null) return null;++ return _syncTx(fn, id);++}++++/** Never-throwing variant for write hooks outside database.js. */++function safeSync(kind, id) {++ try { return sync(kind, id); } catch (err) { console.warn(`[Objects] sync ${kind} ${id}:`, err.message); return null; }++}++++/**++ * After a tier move that verified a copy (upload + HEAD, copy + HEAD): re-project,++ * then mark the verified providers present.++ */++function afterTierMove(vodId, verifiedProviders = []) {++ try {++ const r = sync('vod', vodId);++ if (!r || !r.id) return;++ for (const l of listLocations(r.id)) {++ if (verifiedProviders.includes(l.provider)) setLocationState(l.id, { state: 'present' });++ }++ } catch (err) { console.warn(`[Objects] tier sync vod ${vodId}:`, err.message); }++}++++// ── Holds ────────────────────────────────────────────────────++++function placeHold({ object_id, kind, reason = '', created_by = null }) {++ if (!HOLD_KINDS.includes(kind)) throw new Error(`hold kind must be one of ${HOLD_KINDS.join(', ')}`);++ const r = db.run('INSERT INTO media_holds (object_id, kind, reason, created_by) VALUES (?, ?, ?, ?)',++ [object_id, kind, String(reason || '').slice(0, 1000), created_by]);++ return db.get('SELECT * FROM media_holds WHERE id = ?', [r.lastInsertRowid]);++}++++function releaseHold(holdId, releasedBy = null) {++ db.run('UPDATE media_holds SET released_at = CURRENT_TIMESTAMP, released_by = ? WHERE id = ? AND released_at IS NULL', [releasedBy, holdId]);++ return db.get('SELECT * FROM media_holds WHERE id = ?', [holdId]);++}++++// ── Native objects: soft delete, restore, purge, quota ───────++++function objectFilePath(obj) {++ return path.join(config.objects.path, obj.app_id, obj.id);++}++++function softDelete(obj, { by = null } = {}) {++ if (isHeld(obj.id)) throw new HeldError(obj.id);++ const md = parseJson(obj.metadata, {});++ md.pre_delete_status = obj.lifecycle_status;++ md.retention_until = new Date(Date.now() + config.objects.retentionDays * 864e5).toISOString();++ if (by) md.deleted_by = by;++ db.run(`UPDATE media_objects SET lifecycle_status = 'deleted', deleted_at = CURRENT_TIMESTAMP, metadata = ?, updated_at = CURRENT_TIMESTAMP++ WHERE id = ?`, [JSON.stringify(md), obj.id]);++ return getObject(obj.id);++}++++function restore(obj) {++ const md = parseJson(obj.metadata, {});++ if (md.purged_at) return null;++ const back = md.pre_delete_status && md.pre_delete_status !== 'deleted' ? md.pre_delete_status : 'ready';++ delete md.pre_delete_status; delete md.retention_until; delete md.deleted_by;++ db.run(`UPDATE media_objects SET lifecycle_status = ?, deleted_at = NULL, metadata = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ?`,++ [back, JSON.stringify(md), obj.id]);++ return getObject(obj.id);++}++++/**++ * Remove the bytes of native (v2) objects deleted longer ago than the retention++ * period. Projected objects are never purged here — their bytes belong to the++ * inherited pipelines. Held objects are skipped.++ */++function purgeExpired({ retentionDays = config.objects.retentionDays } = {}) {++ const rows = db.all(`SELECT * FROM media_objects WHERE lifecycle_status = 'deleted' AND legacy_ref IS NULL++ AND deleted_at <= datetime('now', ?)`, [`-${Math.max(0, retentionDays)} days`]);++ const root = path.resolve(config.objects.path) + path.sep;++ let purged = 0;++ for (const obj of rows) {++ const md = parseJson(obj.metadata, {});++ if (md.purged_at || isHeld(obj.id)) continue;++ for (const l of listLocations(obj.id)) {++ if (l.provider === 'local' && path.resolve(l.key).startsWith(root)) { try { fs.unlinkSync(l.key); } catch { /* already gone */ } }++ if (l.provider === 'local') db.run('DELETE FROM media_locations WHERE id = ?', [l.id]);++ }++ md.purged_at = new Date().toISOString();++ db.run('UPDATE media_objects SET metadata = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ?', [JSON.stringify(md), obj.id]);++ purged++;++ }++ return purged;++}++++/** Bytes counted against apps.quota_bytes: v1 files + live native objects (reservations included). */++function usedBytes(appId, excludeId = null) {++ const native = db.get(`SELECT COALESCE(SUM(size_bytes), 0) AS b FROM media_objects++ WHERE app_id = ? AND legacy_ref IS NULL AND lifecycle_status IN ('uploading', 'ready') AND id != ?`,++ [appId, excludeId || '']).b;++ return db.appFilesBytes(appId) + native;++}++++// ── Public shape ─────────────────────────────────────────────++++/** Where a projected object is served today (the inherited public URL), or null. */++function legacyPublicUrl(obj) {++ const ref = parseLegacyRef(obj.legacy_ref);++ if (!ref) return null;++ const id = encodeURIComponent(ref.id);++ const p = { vod: `/v/${id}`, clip: `/c/${id}`, file: `/f/${id}`, thumbnail: `/t/${id}`, paste: `/p/${id}/screenshot`, avatar: `/p/${id}/screenshot` }[ref.kind];++ return p ? config.publicUrl + p : null;++}++++function objectPublic(obj, { locations = true } = {}) {++ if (!obj) return null;++ const out = {++ id: obj.id,++ media_ref: { media_id: obj.id },++ legacy_ref: obj.legacy_ref || null,++ app_id: obj.app_id,++ namespace: obj.namespace,++ kind: obj.kind,++ owner: { subject: obj.owner_subject || null, app: obj.owner_app || null, user_id: obj.owner_user_id ?? null },++ visibility: obj.visibility,++ lifecycle_status: obj.lifecycle_status,++ mime_type: obj.mime_type,++ size_bytes: obj.size_bytes,++ content_hash: obj.content_hash || null,++ metadata: parseJson(obj.metadata, {}),++ held: isHeld(obj.id),++ public_url: obj.visibility !== 'private' && obj.lifecycle_status === 'ready'++ ? (legacyPublicUrl(obj) || `${config.publicUrl}/o/${obj.id}`) : null,++ created_at: obj.created_at,++ updated_at: obj.updated_at,++ deleted_at: obj.deleted_at || null,++ };++ // Where the bytes are, never the keys or paths themselves.++ if (locations) {++ out.locations = listLocations(obj.id).map(l => ({++ provider: l.provider, storage_class: l.storage_class, state: l.state, size_bytes: l.size_bytes,++ verified_at: l.verified_at, canonical: l.provider === obj.canonical_provider,++ }));++ }++ return out;++}++++module.exports = {++ KINDS, VISIBILITIES, LIFECYCLES, HOLD_KINDS, HeldError,++ mimeFor, parseJson, parseLegacyRef, thumbFileFromUrl,++ getObject, getObjectByLegacyRef, resolveObject, listLocations, listHolds, isHeld, isHeldRow,++ createObject, updateObject, upsertLocation, setLocationState, setRelationship, setVariant, getVariant,++ syncVod, syncClip, syncFile, syncPaste, sync, safeSync, afterTierMove,++ placeHold, releaseHold,++ objectFilePath, softDelete, restore, purgeExpired, usedBytes,++ legacyPublicUrl, objectPublic,++};nnew file mode 100644@@@ -0,0 +1,197 @@++/**++ * OpenVibe.Media — object reconciliation (docs/object-model.md#reconciliation)++ *++ * Compares the object model with the bytes. Read-only by default: it checks++ * local files and the database and reports. With verify=true it also HEADs++ * every B2/R2 location (through the storage engine's S3 client, or an injected++ * `head` for tests) and writes what it learned back to media_locations++ * (state present/missing/corrupt, size, verified_at). Never deletes anything.++ *++ * Detects:++ * no_canonical_location ready object with no row for its canonical copy++ * canonical_missing_replica_present canonical copy gone/corrupt while another copy is present++ * no_present_copy ready object with no present copy anywhere++ * missing_local_file ready object's local location whose file is not on disk++ * remote_missing ready object's B2/R2 location whose HEAD says 404 (verify)++ * size_mismatch / hash_mismatch bytes differ from the object's size / sha256++ * orphan_location location rows whose object does not exist++ * deleted_publicly_reachable deleted object still served (legacy row, or a thumbnail of it)++ * missing_projection inherited rows with no object yet (run the backfill)++ */++'use strict';++++const fs = require('fs');++const crypto = require('crypto');++const db = require('../db/database');++const model = require('./model');++++const MB = 1024 * 1024;++const ISSUES = ['no_canonical_location', 'canonical_missing_replica_present', 'no_present_copy', 'missing_local_file', 'remote_missing',++ 'size_mismatch', 'hash_mismatch', 'orphan_location', 'deleted_publicly_reachable', 'missing_projection'];++++/** Real provider: undefined when the provider is not configured here (cannot verify), null on 404. */++async function defaultHead(provider, key) {++ const vs = require('../vod/vod-storage');++ if (!vs.providerConfigured(provider)) return undefined;++ return vs.headObject(provider, key);++}++++function sha256File(p) {++ return new Promise((resolve, reject) => {++ const h = crypto.createHash('sha256');++ const s = fs.createReadStream(p);++ s.on('data', d => h.update(d));++ s.on('end', () => resolve(h.digest('hex')));++ s.on('error', reject);++ });++}++++// Which inherited row serves a projected object, and is it reachable without credentials?++const PROJECTIONS = [++ { table: 'vods', open: (r) => (r.visibility || (r.is_public ? 'public' : 'private')) !== 'private' && !r.clips_only },++ { table: 'clips', open: (r) => (r.visibility || (r.is_public ? 'public' : 'private')) !== 'private' },++ { table: 'files', open: () => true },++ { table: 'pastes', open: (r) => r.visibility !== 'private' && !!r.screenshot_path },++];++++async function reconcile({ verify = false, hash = false, head = defaultHead, hashMaxBytes = 512 * MB, appId = null, listLimit = 200 } = {}) {++ const report = {++ generated_at: new Date().toISOString(),++ mode: { verify: !!verify, hash: !!hash, app_id: appId },++ counts: { objects: 0, locations: 0, local_checked: 0, remote_checked: 0, remote_unverified: 0, remote_unverifiable: 0, locations_updated: 0 },++ issues: Object.fromEntries(ISSUES.map(k => [k, { count: 0, items: [] }])),++ };++ const issue = (k, item) => { const i = report.issues[k]; i.count++; if (i.items.length < listLimit) i.items.push(item); };++++ const objects = db.all(`SELECT * FROM media_objects${appId ? ' WHERE app_id = ?' : ''}`, appId ? [appId] : []);++ const byId = new Map(objects.map(o => [o.id, o]));++ const locations = appId++ ? db.all('SELECT l.* FROM media_locations l LEFT JOIN media_objects o ON o.id = l.object_id WHERE o.app_id = ? OR o.id IS NULL', [appId])++ : db.all('SELECT * FROM media_locations');++ report.counts.objects = objects.length;++ report.counts.locations = locations.length;++++ const effective = new Map(); // location id → state after this run's checks++ const byObject = new Map();++ const write = (loc, state, size) => {++ if (!verify) return;++ model.setLocationState(loc.id, { state, size_bytes: size });++ report.counts.locations_updated++;++ };++++ for (const loc of locations) {++ const obj = byId.get(loc.object_id);++ if (!obj) { issue('orphan_location', { location_id: loc.id, object_id: loc.object_id, provider: loc.provider }); continue; }++ if (!byObject.has(obj.id)) byObject.set(obj.id, []);++ byObject.get(obj.id).push(loc);++ // Missing bytes are an issue for ready objects; a recording or upload may not have written yet.++ const live = obj.lifecycle_status === 'ready';++ const expected = Number(obj.size_bytes) || 0;++ const sizeMatters = obj.lifecycle_status === 'ready' && expected > 0;++++ if (loc.provider === 'local') {++ report.counts.local_checked++;++ let st = null;++ try { const s = fs.statSync(loc.key); if (s.isFile()) st = s; } catch { /* missing */ }++ if (!st) {++ effective.set(loc.id, 'missing');++ if (live) issue('missing_local_file', { object_id: obj.id, kind: obj.kind, legacy_ref: obj.legacy_ref, path: loc.key });++ write(loc, 'missing', null);++ continue;++ }++ let state = 'present';++ if (sizeMatters && st.size !== expected) {++ issue('size_mismatch', { object_id: obj.id, provider: 'local', expected, actual: st.size, legacy_ref: obj.legacy_ref });++ }++ if (hash && obj.content_hash && st.size <= hashMaxBytes) {++ const actual = await sha256File(loc.key).catch(() => null);++ if (actual && actual !== obj.content_hash) {++ state = 'corrupt';++ issue('hash_mismatch', { object_id: obj.id, provider: 'local', expected: obj.content_hash, actual });++ }++ }++ effective.set(loc.id, state);++ write(loc, state, st.size);++ continue;++ }++++ // b2 / r2++ if (!verify) {++ effective.set(loc.id, loc.state);++ if (loc.state === 'pending') report.counts.remote_unverified++;++ continue;++ }++ let h;++ try { h = await head(loc.provider, loc.key, loc.bucket); } catch { h = undefined; report.counts.remote_unverifiable++; effective.set(loc.id, loc.state); continue; }++ if (h === undefined) { report.counts.remote_unverifiable++; effective.set(loc.id, loc.state); continue; }++ report.counts.remote_checked++;++ if (!h) {++ effective.set(loc.id, 'missing');++ if (live) issue('remote_missing', { object_id: obj.id, provider: loc.provider, key: loc.key, legacy_ref: obj.legacy_ref });++ write(loc, 'missing', null);++ continue;++ }++ let state = 'present';++ if (sizeMatters && Number(h.size) !== expected) {++ state = 'corrupt';++ issue('size_mismatch', { object_id: obj.id, provider: loc.provider, expected, actual: Number(h.size), legacy_ref: obj.legacy_ref });++ }++ effective.set(loc.id, state);++ write(loc, state, Number(h.size));++ }++++ // Per object: is the canonical copy there, and is any copy there at all?++ for (const obj of objects) {++ if (obj.lifecycle_status !== 'ready') continue;++ const locs = byObject.get(obj.id) || [];++ const canon = obj.canonical_provider ? locs.find(l => l.provider === obj.canonical_provider) : null;++ const present = locs.filter(l => effective.get(l.id) === 'present');++ if (!canon) {++ issue('no_canonical_location', { object_id: obj.id, kind: obj.kind, canonical_provider: obj.canonical_provider, legacy_ref: obj.legacy_ref });++ } else if (['missing', 'corrupt'].includes(effective.get(canon.id))) {++ if (present.length) issue('canonical_missing_replica_present', { object_id: obj.id, canonical: canon.provider, present: present.map(l => l.provider), legacy_ref: obj.legacy_ref });++ }++ const knownGone = locs.length && locs.every(l => ['missing', 'corrupt'].includes(effective.get(l.id)));++ if (knownGone) issue('no_present_copy', { object_id: obj.id, kind: obj.kind, legacy_ref: obj.legacy_ref });++ }++++ // Deleted objects that the inherited routes still serve.++ for (const p of PROJECTIONS) {++ const rows = db.all(`SELECT t.* FROM ${p.table} t JOIN media_objects o ON o.id = t.object_id WHERE o.lifecycle_status = 'deleted'${appId ? ' AND o.app_id = ?' : ''}`, appId ? [appId] : []);++ for (const r of rows) if (p.open(r)) issue('deleted_publicly_reachable', { object_id: r.object_id, table: p.table, id: r.id ?? r.key, reason: 'the inherited row still serves it' });++ }++ // …and derivatives (thumbnails) of deleted objects that are still served by name.++ const orphanThumbs = db.all(`SELECT d.* FROM media_variants v JOIN media_objects p ON p.id = v.object_id JOIN media_objects d ON d.id = v.derived_object_id++ WHERE p.lifecycle_status = 'deleted' AND d.lifecycle_status != 'deleted' AND d.visibility != 'private'${appId ? ' AND p.app_id = ?' : ''}`, appId ? [appId] : []);++ for (const d of orphanThumbs) {++ const loc = (byObject.get(d.id) || []).find(l => effective.get(l.id) === 'present');++ if (loc) issue('deleted_publicly_reachable', { object_id: d.id, kind: d.kind, legacy_ref: d.legacy_ref, reason: 'derivative of a deleted object is still served' });++ }++++ // Inherited rows the model does not cover yet.++ const gaps = [++ ['vods', 'id', 'object_id IS NULL AND COALESCE(clips_only, 0) = 0'],++ ['clips', 'id', 'object_id IS NULL'],++ ['files', 'key', 'object_id IS NULL'],++ ['pastes', 'id', "object_id IS NULL AND type = 'screenshot' AND screenshot_path IS NOT NULL"],++ ];++ for (const [table, keyCol, cond] of gaps) {++ for (const r of db.all(`SELECT ${keyCol} AS k FROM ${table} WHERE ${cond}${appId ? ' AND app_id = ?' : ''}`, appId ? [appId] : [])) {++ issue('missing_projection', { table, id: r.k });++ }++ }++++ report.total_issues = ISSUES.reduce((n, k) => n + report.issues[k].count, 0);++ return report;++}++++function summarize(report) {++ const c = report.counts;++ const lines = [`objects ${c.objects}, locations ${c.locations} (local checked ${c.local_checked}, remote checked ${c.remote_checked}, `++ + `remote unverified ${c.remote_unverified}, unverifiable ${c.remote_unverifiable}${report.mode.verify ? `, updated ${c.locations_updated}` : ''})`];++ for (const k of ISSUES) if (report.issues[k].count) lines.push(` ${k}: ${report.issues[k].count}`);++ if (!report.total_issues) lines.push(' no issues');++ return lines.join('\n');++}++++module.exports = { reconcile, summarize, ISSUES, defaultHead };nnew file mode 100644@@@ -0,0 +1,409 @@++/**++ * OpenVibe.Media — Object API v2 (mounted at /api/v2/:app/objects; docs/object-model.md)++ *++ * POST / init upload → { id, object, upload: { method, url, token, expires_at, max_bytes } }++ * PUT /:id/content the bytes (single part; sha256 computed; size + quota checked)++ * POST /:id/complete verify (expected hash, invariant) → lifecycle ready++ * GET /:id metadata (id = med_… or legacy:<app>:<kind>:<id>)++ * GET / cursor list (?kind&visibility&status&owner&user_id&include_deleted&limit&cursor)++ * DELETE /:id soft delete (bytes kept MEDIA_DELETE_RETENTION_DAYS); 409 when held++ * POST /:id/restore undo a soft delete inside the retention period++ * GET /:id/download 302 to the public location, or a signed short-lived URL for private objects++ * GET /:id/holds retention holds (?all=1 includes released)++ * POST /:id/holds place a hold { kind, reason, created_by } (app key only)++ * DELETE /:id/holds/:holdId release it (app key only)++ *++ * Auth: the app's API key, or a Network service token granting media.object.upload++ * (writes) / media.object.read (reads) for namespace = :app. X-OV-Subject names the++ * owner (usr_…); X-OV-User-Id (app key only) acts as one of the app's users. The++ * content PUT and complete also accept the upload token handed out at init, so a++ * browser can send the bytes directly.++ *++ * Public bytes: GET /o/:id (publicRouter) — public/unlisted objects openly,++ * private ones only with a valid ?exp&sig from /download.++ */++'use strict';++++const express = require('express');++const fs = require('fs');++const path = require('path');++const crypto = require('crypto');++const { ids, http } = require('openvibe-contracts');++const db = require('../db/database');++const config = require('../config');++const model = require('./model');++const invariant = require('./invariant');++const signing = require('./signing');++const { tenantAuth, tenantCors } = require('../auth');++const { sendWebhook } = require('../webhooks');++++const MB = 1024 * 1024;++const upload = tenantAuth({ capability: 'media.object.upload' });++const read = tenantAuth({ capability: 'media.object.read' });++const appOnly = tenantAuth();++++function problem(res, status, code, detail, extra) {++ return http.sendProblem(res, status, code, { detail, extra });++}++++/** Owner subject from X-OV-Subject: 'usr_…' or 'user:usr_…'. undefined = malformed. */++function subjectFrom(req) {++ const raw = req.headers['x-ov-subject'];++ if (raw == null || raw === '') return null;++ const s = String(raw).trim();++ const parsed = ids.parseSubject(s);++ if (parsed && parsed.type === 'user') return parsed.id;++ return ids.isSubjectId('user', s) ? s : undefined;++}++++// An app acting for one of its users (X-OV-User-Id) sees public/unlisted objects++// plus that user's own; the app itself and service principals see the namespace.++function canSee(req, obj) {++ return req.authType !== 'user' || obj.visibility !== 'private' || (obj.owner_user_id != null && obj.owner_user_id === req.userId);++}++function canWrite(req, obj) {++ return req.authType !== 'user' || (obj.owner_user_id != null && obj.owner_user_id === req.userId);++}++++function load(req, res) {++ const obj = model.resolveObject(String(req.params.id || ''), req.appId);++ if (!obj || !canSee(req, obj)) { problem(res, 404, 'media.object.not_found', 'No such object in this namespace'); return null; }++ return obj;++}++++/** Upload token (from init) or the usual tenant credential with media.object.upload. */++function contentAuth(req, res, next) {++ const token = req.query.token || req.headers['x-upload-token'];++ if (!token) return upload(req, res, next);++ const app = db.getApp(String(req.params.app || ''));++ if (!app) return res.status(404).json({ error: 'Unknown app' });++ if (!signing.verifyUploadToken(String(req.params.id || ''), token)) return problem(res, 401, 'media.upload_token.invalid', 'Upload token is invalid or expired');++ req.appId = app.app_id;++ req.appRow = app;++ req.authType = 'upload_token';++ next();++}++++function sanitizeFilename(name) {++ if (name == null || name === '') return null;++ return path.basename(String(name)).replace(/[^\w.\- ]/g, '_').slice(0, 200) || null;++}++++function quotaCheck(req, res, addBytes, excludeId = null) {++ const quota = Number(req.appRow.quota_bytes) || 0;++ if (!quota) return true;++ const used = model.usedBytes(req.appId, excludeId);++ if (used + addBytes <= quota) return true;++ problem(res, 413, 'media.quota.exceeded', 'App storage quota exceeded', { quota_bytes: quota, used_bytes: used });++ return false;++}++++// ── Content upload (mounted ahead of the JSON body parser — see index.js) ──++++/** Stream the request body to `tmp`, hashing as it goes; stop past `cap` bytes. */++function receive(req, tmp, cap) {++ return new Promise((resolve) => {++ const hash = crypto.createHash('sha256');++ const out = fs.createWriteStream(tmp);++ let n = 0, done = false;++ const finish = (r) => { if (!done) { done = true; resolve(r); } };++ // Failures settle once the file handle is closed, so the caller's unlink cannot race the open.++ const fail = (r) => { if (done) return; if (out.closed) return finish(r); out.once('close', () => finish(r)); out.destroy(); };++ req.on('data', (chunk) => {++ if (done) return;++ n += chunk.length;++ if (n > cap) { req.unpipe(out); req.resume(); return fail({ error: 'too_large' }); }++ hash.update(chunk);++ });++ req.on('aborted', () => fail({ error: 'aborted' }));++ req.on('error', () => fail({ error: 'aborted' }));++ out.on('error', (e) => fail({ error: 'write', message: e.message }));++ out.on('finish', () => finish({ bytes: n, sha256: hash.digest('hex') }));++ req.pipe(out);++ });++}++++async function putContent(req, res) {++ const obj = load(req, res);++ if (!obj) { req.resume(); return; }++ if (!canWrite(req, obj)) { req.resume(); return problem(res, 403, 'media.object.forbidden', 'Not your object'); }++ if (obj.legacy_ref) { req.resume(); return problem(res, 409, 'media.object.legacy_managed', 'This object is written through the v1 API'); }++ if (obj.lifecycle_status !== 'uploading') { req.resume(); return problem(res, 409, 'media.object.not_uploading', `Object is ${obj.lifecycle_status}`); }++++ const declared = Number(obj.size_bytes) > 0 ? Number(obj.size_bytes) : null;++ const cap = declared || config.objects.maxUploadMb * MB;++ const len = Number(req.headers['content-length']);++ if (Number.isFinite(len) && len > cap) {++ res.set('Connection', 'close'); req.resume();++ return problem(res, 413, 'media.object.too_large', `At most ${cap} bytes`);++ }++++ const tmpDir = path.join(config.objects.path, '.tmp');++ fs.mkdirSync(tmpDir, { recursive: true });++ const tmp = path.join(tmpDir, `${obj.id}-${crypto.randomBytes(4).toString('hex')}`);++ const got = await receive(req, tmp, cap);++ const drop = () => { try { fs.unlinkSync(tmp); } catch { /* not written */ } };++ if (got.error) {++ drop();++ if (got.error === 'too_large') { res.set('Connection', 'close'); return problem(res, 413, 'media.object.too_large', `At most ${cap} bytes`); }++ return problem(res, 400, 'media.object.upload_interrupted', got.message || 'Upload did not complete');++ }++ if (declared && got.bytes !== declared) { drop(); return problem(res, 400, 'media.object.size_mismatch', `Declared ${declared} bytes, received ${got.bytes}`); }++ if (!got.bytes) { drop(); return problem(res, 400, 'media.object.empty', 'No bytes received'); }++ if (!quotaCheck(req, res, got.bytes, obj.id)) { drop(); return; }++++ try {++ const dest = model.objectFilePath(obj);++ fs.mkdirSync(path.dirname(dest), { recursive: true });++ fs.renameSync(tmp, dest);++ const contentType = String(req.headers['content-type'] || '').split(';')[0].trim().toLowerCase();++ db.getDb().transaction(() => {++ model.updateObject(obj.id, {++ size_bytes: got.bytes, content_hash: got.sha256,++ mime_type: obj.mime_type || (/^[\w.+-]+\/[\w.+-]+$/.test(contentType) ? contentType : 'application/octet-stream'),++ canonical_provider: 'local', canonical_key: dest,++ });++ model.upsertLocation(obj.id, { provider: 'local', key: dest, state: 'present', size_bytes: got.bytes, checksum: got.sha256, verified: true });++ })();++ res.json({ id: obj.id, size_bytes: got.bytes, content_hash: got.sha256 });++ } catch (err) {++ drop();++ console.error('[Objects] content store error:', err.message);++ problem(res, 500, 'media.object.store_failed', 'Failed to store content');++ }++}++++// ── API router ───────────────────────────────────────────────++++const router = express.Router({ mergeParams: true });++router.use(tenantCors);++++router.post('/', upload, (req, res) => {++ try {++ const b = req.body || {};++ const kind = String(b.kind || 'file');++ if (!model.KINDS.includes(kind)) return problem(res, 400, 'media.object.invalid', `kind must be one of ${model.KINDS.join(', ')}`);++ const visibility = String(b.visibility || 'private');++ if (!model.VISIBILITIES.includes(visibility)) return problem(res, 400, 'media.object.invalid', 'visibility must be public, unlisted or private');++ let size = 0;++ if (b.size_bytes != null) {++ size = Number(b.size_bytes);++ if (!Number.isInteger(size) || size < 1) return problem(res, 400, 'media.object.invalid', 'size_bytes must be a positive integer');++ }++ const max = config.objects.maxUploadMb * MB;++ if (size > max) return problem(res, 413, 'media.object.too_large', `Single-part uploads are limited to ${max} bytes`);++ if (size && invariant.wouldViolate({ kind, visibility, size_bytes: size })) {++ return problem(res, 422, 'media.invariant.public_object_too_large', `Public playback objects are limited to ${config.objects.publicMaxMb} MB`);++ }++ if (b.content_hash != null && !/^[a-f0-9]{64}$/i.test(String(b.content_hash))) return problem(res, 400, 'media.object.invalid', 'content_hash must be a sha256 hex digest');++ if (b.metadata != null && (typeof b.metadata !== 'object' || Array.isArray(b.metadata) || JSON.stringify(b.metadata).length > 16384)) {++ return problem(res, 400, 'media.object.invalid', 'metadata must be an object of at most 16 KB');++ }++ const subject = subjectFrom(req);++ if (subject === undefined) return problem(res, 400, 'media.object.invalid', 'X-OV-Subject must be a user subject (usr_…)');++ if (!quotaCheck(req, res, size)) return;++++ const mime = b.mime_type && /^[\w.+-]+\/[\w.+-]+$/.test(String(b.mime_type)) ? String(b.mime_type).toLowerCase() : null;++ const userId = req.authType === 'user' ? req.userId : (Number.isInteger(Number(b.user_id)) && Number(b.user_id) > 0 ? Number(b.user_id) : null);++ const metadata = { ...(b.metadata || {}) };++ const filename = sanitizeFilename(b.filename);++ if (filename) metadata.filename = filename;++ if (b.content_hash) metadata.expected_sha256 = String(b.content_hash).toLowerCase();++ if (req.principal) metadata.created_by = req.principal.sub;++++ const id = model.createObject({++ app_id: req.appId, namespace: req.appId, kind, owner_subject: subject, owner_app: req.appId, owner_user_id: userId,++ visibility, lifecycle_status: 'uploading', mime_type: mime, size_bytes: size, metadata,++ });++ const tok = signing.uploadToken(id);++ res.status(201).json({++ id,++ object: model.objectPublic(model.getObject(id)),++ upload: {++ method: 'PUT',++ url: `${config.publicUrl}/api/v2/${encodeURIComponent(req.appId)}/objects/${id}/content?token=${encodeURIComponent(tok.token)}`,++ token: tok.token,++ expires_at: tok.expires_at,++ max_bytes: size || max,++ complete_url: `${config.publicUrl}/api/v2/${encodeURIComponent(req.appId)}/objects/${id}/complete`,++ },++ });++ } catch (err) {++ console.error('[Objects] init error:', err.message);++ problem(res, 500, 'media.object.init_failed', 'Failed to create object');++ }++});++++router.post('/:id/complete', contentAuth, tenantCors, (req, res) => {++ try {++ const obj = load(req, res);++ if (!obj) return;++ if (!canWrite(req, obj)) return problem(res, 403, 'media.object.forbidden', 'Not your object');++ if (obj.lifecycle_status === 'ready' && !obj.legacy_ref) return res.json(model.objectPublic(obj));++ if (obj.legacy_ref || obj.lifecycle_status !== 'uploading') return problem(res, 409, 'media.object.not_uploading', `Object is ${obj.lifecycle_status}`);++ const loc = model.listLocations(obj.id).find(l => l.provider === 'local' && l.state === 'present');++ if (!loc || !fs.existsSync(loc.key)) return problem(res, 409, 'media.object.no_content', 'PUT the content first');++ const md = model.parseJson(obj.metadata, {});++ const expected = String((req.body && req.body.content_hash) || md.expected_sha256 || '').toLowerCase();++ if (expected && expected !== obj.content_hash) {++ return problem(res, 422, 'media.object.hash_mismatch', 'Content hash does not match', { expected, actual: obj.content_hash });++ }++ if (invariant.wouldViolate(obj)) {++ return problem(res, 422, 'media.invariant.public_object_too_large', `Public playback objects are limited to ${config.objects.publicMaxMb} MB`);++ }++ if (!quotaCheck(req, res, Number(obj.size_bytes) || 0, obj.id)) return;++ model.updateObject(obj.id, { lifecycle_status: 'ready' });++ const done = model.getObject(obj.id);++ invariant.record(done);++ const body = model.objectPublic(done);++ sendWebhook(req.appId, 'media.object.uploaded', body).catch(() => {});++ res.json(body);++ } catch (err) {++ console.error('[Objects] complete error:', err.message);++ problem(res, 500, 'media.object.complete_failed', 'Failed to complete upload');++ }++});++++router.get('/', read, (req, res) => {++ try {++ const q = req.query;++ const limit = Math.min(Math.max(parseInt(q.limit || '50', 10) || 50, 1), 200);++ const conds = ['app_id = ?'], params = [req.appId];++ if (q.cursor) {++ if (!/^med_[0-9A-HJKMNP-TV-Z]{26}$/.test(String(q.cursor))) return problem(res, 400, 'media.object.invalid', 'Bad cursor');++ conds.push('id < ?'); params.push(String(q.cursor));++ }++ if (q.kind) { conds.push('kind = ?'); params.push(String(q.kind)); }++ if (q.visibility) { conds.push('visibility = ?'); params.push(String(q.visibility)); }++ if (q.status) { conds.push('lifecycle_status = ?'); params.push(String(q.status)); }++ else if (!['1', 'true'].includes(String(q.include_deleted || ''))) conds.push("lifecycle_status != 'deleted'");++ if (q.owner) { conds.push('owner_subject = ?'); params.push(String(q.owner).replace(/^user:/, '')); }++ if (q.user_id != null) { conds.push('owner_user_id = ?'); params.push(Number(q.user_id)); }++ if (req.authType === 'user') { conds.push("(visibility != 'private' OR owner_user_id = ?)"); params.push(req.userId); }++ const rows = db.all(`SELECT * FROM media_objects WHERE ${conds.join(' AND ')} ORDER BY id DESC LIMIT ?`, [...params, limit + 1]);++ const page = rows.slice(0, limit);++ res.json({ objects: page.map(o => model.objectPublic(o, { locations: false })), next_cursor: rows.length > limit ? page[page.length - 1].id : null, limit });++ } catch (err) {++ console.error('[Objects] list error:', err.message);++ problem(res, 500, 'media.object.list_failed', 'Failed to list objects');++ }++});++++router.get('/:id', read, (req, res) => {++ const obj = load(req, res);++ if (obj) res.json(model.objectPublic(obj));++});++++router.delete('/:id', upload, (req, res) => {++ try {++ const obj = load(req, res);++ if (!obj) return;++ if (!canWrite(req, obj)) return problem(res, 403, 'media.object.forbidden', 'Not your object');++ if (obj.legacy_ref) return problem(res, 409, 'media.object.legacy_managed', 'Delete this object through its v1 route (vods, clips, files, pastes)');++ if (model.isHeld(obj.id)) return problem(res, 409, 'media.object.held', 'Object is under a retention hold');++ if (obj.lifecycle_status === 'deleted') return res.json(model.objectPublic(obj));++ res.json(model.objectPublic(model.softDelete(obj, { by: req.principal ? req.principal.sub : `app:${req.appId}` })));++ } catch (err) {++ if (err.code === 'media.object.held' || /retention hold/.test(err.message)) return problem(res, 409, 'media.object.held', 'Object is under a retention hold');++ console.error('[Objects] delete error:', err.message);++ problem(res, 500, 'media.object.delete_failed', 'Failed to delete object');++ }++});++++router.post('/:id/restore', upload, (req, res) => {++ const obj = load(req, res);++ if (!obj) return;++ if (!canWrite(req, obj)) return problem(res, 403, 'media.object.forbidden', 'Not your object');++ if (obj.lifecycle_status !== 'deleted' || obj.legacy_ref) return problem(res, 409, 'media.object.not_deleted', 'Only soft-deleted native objects can be restored');++ const back = model.restore(obj);++ if (!back) return problem(res, 410, 'media.object.purged', 'The retention period has passed and the bytes are gone');++ res.json(model.objectPublic(back));++});++++router.get('/:id/download', read, (req, res) => {++ const obj = load(req, res);++ if (!obj) return;++ if (obj.lifecycle_status === 'deleted') return problem(res, 410, 'media.object.deleted', 'Object was deleted');++ if (obj.lifecycle_status !== 'ready') return problem(res, 409, 'media.object.not_ready', `Object is ${obj.lifecycle_status}`);++ const json = req.query.format === 'json';++ res.set('Cache-Control', 'private, no-store');++ if (obj.visibility !== 'private') {++ const url = model.legacyPublicUrl(obj) || `${config.publicUrl}/o/${obj.id}`;++ return json ? res.json({ url, expires_at: null, public: true }) : res.redirect(302, url);++ }++ const signed = signing.signedDownloadUrl(obj.id, req.query.ttl);++ if (['1', 'true'].includes(String(req.query.redirect || ''))) return res.redirect(302, signed.url);++ res.json({ url: signed.url, expires_at: signed.expires_at, public: false });++});++++// ── Retention holds ──────────────────────────────────────────++++router.get('/:id/holds', read, (req, res) => {++ const obj = load(req, res);++ if (!obj) return;++ res.json({ object_id: obj.id, holds: model.listHolds(obj.id, { includeReleased: ['1', 'true'].includes(String(req.query.all || '')) }) });++});++++router.post('/:id/holds', appOnly, (req, res) => {++ const obj = load(req, res);++ if (!obj) return;++ const b = req.body || {};++ if (!model.HOLD_KINDS.includes(b.kind)) return problem(res, 400, 'media.hold.invalid', `kind must be one of ${model.HOLD_KINDS.join(', ')}`);++ const by = b.created_by ? String(b.created_by).slice(0, 200) : `app:${req.appId}${req.userId != null ? `:user:${req.userId}` : ''}`;++ res.status(201).json(model.placeHold({ object_id: obj.id, kind: b.kind, reason: b.reason || '', created_by: by }));++});++++router.delete('/:id/holds/:holdId', appOnly, (req, res) => {++ const obj = load(req, res);++ if (!obj) return;++ const hold = db.get('SELECT * FROM media_holds WHERE id = ? AND object_id = ?', [parseInt(req.params.holdId, 10), obj.id]);++ if (!hold) return problem(res, 404, 'media.hold.not_found', 'No such hold on this object');++ const by = (req.body && req.body.released_by) ? String(req.body.released_by).slice(0, 200) : `app:${req.appId}`;++ res.json(model.releaseHold(hold.id, by));++});++++// ── Public bytes: GET /o/:id ─────────────────────────────────++++const INLINE = /^(image\/(?!svg)|video\/|audio\/|application\/pdf$|text\/plain$)/;++++const publicRouter = express.Router();++publicRouter.get('/:id', async (req, res) => {++ try {++ const obj = model.getObject(String(req.params.id || ''));++ if (!obj) return res.status(404).json({ error: 'Not found' });++ if (obj.lifecycle_status === 'deleted') return res.status(410).json({ error: 'Gone' });++ if (obj.lifecycle_status !== 'ready') return res.status(404).json({ error: 'Not found' });++ const signed = !!req.query.sig && signing.verifyDownload(obj.id, req.query.exp, req.query.sig);++ // Private objects are indistinguishable from missing ones without a valid signature.++ if (obj.visibility === 'private' && !signed) return res.status(404).json({ error: 'Not found' });++++ const mime = obj.mime_type || 'application/octet-stream';++ const md = model.parseJson(obj.metadata, {});++ const name = String(md.filename || obj.id).replace(/["\\\r\n]/g, '_');++ const headers = {++ 'Content-Type': mime,++ 'X-Content-Type-Options': 'nosniff',++ 'X-Robots-Tag': 'noindex',++ 'Cache-Control': obj.visibility === 'private' ? 'private, no-store' : 'public, max-age=3600',++ 'Content-Disposition': `${INLINE.test(mime) ? 'inline' : 'attachment'}; filename="${name}"`,++ };++ const locs = model.listLocations(obj.id);++ const local = locs.find(l => l.provider === 'local' && l.state !== 'missing' && fs.existsSync(l.key));++ if (local) return require('../public/routes').streamFileWithRange(req, res, local.key, headers);++ const vodStorage = require('../vod/vod-storage');++ for (const p of ['r2', 'b2']) {++ const l = locs.find(x => x.provider === p && x.state !== 'missing' && x.state !== 'corrupt');++ if (!l || !vodStorage.providerConfigured(p)) continue;++ const url = await vodStorage.presignGet(p, l.key, 300).catch(() => null);++ if (url) { res.set('Cache-Control', 'private, max-age=0'); res.set('X-Robots-Tag', 'noindex'); return res.redirect(302, url); }++ }++ res.status(404).json({ error: 'Object bytes unavailable' });++ } catch (err) {++ console.error('[Objects] /o error:', err.message);++ if (!res.headersSent) res.status(500).json({ error: 'Failed to serve object' });++ }++});++++module.exports = router;++module.exports.contentHandlers = [contentAuth, tenantCors, putContent];++module.exports.publicRouter = publicRouter;nnew file mode 100644@@@ -0,0 +1,65 @@++/**++ * OpenVibe.Media — signed object URLs and upload tokens++ *++ * HMAC-SHA256 over (purpose, object id, expiry) with MEDIA_SIGNING_SECRET.++ * Purposes are separate so a download signature can never be replayed as an++ * upload token or the other way round. Without the env secret a random++ * per-process secret is used (and a warning logged): links then stop working++ * at the next restart, which is safe but not what production wants.++ */++'use strict';++++const crypto = require('crypto');++const config = require('../config');++++let _ephemeral = null;++function secret() {++ if (config.objects.signingSecret) return config.objects.signingSecret;++ if (!_ephemeral) {++ _ephemeral = crypto.randomBytes(32).toString('hex');++ console.warn('[Objects] MEDIA_SIGNING_SECRET is not set — signed URLs and upload tokens die with this process');++ }++ return _ephemeral;++}++++function mac(purpose, objectId, exp) {++ return crypto.createHmac('sha256', secret()).update(`${purpose}\n${objectId}\n${exp}`).digest('base64url');++}++++function safeEqual(a, b) {++ const x = Buffer.from(String(a || '')), y = Buffer.from(String(b || ''));++ return x.length === y.length && x.length > 0 && crypto.timingSafeEqual(x, y);++}++++function nowS() { return Math.floor(Date.now() / 1000); }++++/** Short-lived GET URL for /o/:id. ttl is clamped to [30 s, 1 h]. */++function signedDownloadUrl(objectId, ttlS = config.objects.signedUrlTtlS) {++ const exp = nowS() + Math.min(3600, Math.max(30, Number(ttlS) || config.objects.signedUrlTtlS));++ return {++ url: `${config.publicUrl}/o/${encodeURIComponent(objectId)}?exp=${exp}&sig=${mac('get', objectId, exp)}`,++ expires_at: new Date(exp * 1000).toISOString(),++ };++}++++function verifyDownload(objectId, exp, sig) {++ const e = Number(exp);++ if (!Number.isInteger(e) || e < nowS()) return false;++ return safeEqual(sig, mac('get', objectId, e));++}++++/** Token for PUT /api/v2/:app/objects/:id/content: "<exp>.<mac>". */++function uploadToken(objectId, ttlS = config.objects.uploadTokenTtlS) {++ const exp = nowS() + Math.max(60, Number(ttlS) || 3600);++ return { token: `${exp}.${mac('put', objectId, exp)}`, expires_at: new Date(exp * 1000).toISOString() };++}++++function verifyUploadToken(objectId, token) {++ const m = /^(\d+)\.([A-Za-z0-9_-]+)$/.exec(String(token || ''));++ if (!m) return false;++ const exp = Number(m[1]);++ if (exp < nowS()) return false;++ return safeEqual(m[2], mac('put', objectId, exp));++}++++module.exports = { signedDownloadUrl, verifyDownload, uploadToken, verifyUploadToken };@@@ -247,6 +247,9 @@ function _getPasteScoped(req, res) {// Fully remove a paste's screenshot from local disk AND any legacy B2 object.function removePasteScreenshot(paste) {if (!paste) return;++ // A held screenshot keeps its bytes (the row delete that follows is refused by the hold trigger).++ const objects = require('../objects/model');++ if (objects.isHeldRow(paste.object_id !== undefined ? paste : db.get('SELECT object_id FROM pastes WHERE id = ?', [paste.id]))) return;if (paste.screenshot_path) {try { fs.unlinkSync(paste.screenshot_path); } catch { /* ignore */ }}@@@ -413,6 +416,7 @@ router.post('/:slug/censor', tenantAuth(), (req, res) => {if (paste.type !== 'screenshot' || !paste.screenshot_path) {return res.status(400).json({ error: 'Not a screenshot paste' });}++ if (require('../objects/model').isHeldRow(paste)) return res.status(409).json({ error: 'Screenshot is under a retention hold', code: 'media.object.held' });const censorUpload = multer({storage: screenshotStorage,@@@ -432,6 +436,7 @@ router.post('/:slug/censor', tenantAuth(), (req, res) => {try { fs.unlinkSync(paste.screenshot_path); } catch { /* */ }db.run('UPDATE pastes SET screenshot_path = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ?',[req.file.path, paste.id]);++ require('../objects/model').safeSync('paste', paste.id);res.json({ paste: pastePublic(db.getPasteBySlug(paste.slug, req.appId)) });} catch (err2) {console.error('[Pastes] Censor error:', err2.message);@@@ -526,6 +531,7 @@ router.post('/', tenantAuth({ allowUser: true }), screenshotUpload.single('scree);const paste = db.getPasteBySlug(slug, req.appId);++ require('../objects/model').safeSync('paste', paste.id);return res.status(201).json({ id: paste.id, slug, url: `/p/${slug}`, paste: pastePublic(paste) });}@@@ -513,3 +513,4 @@ router.get('/p/:slug/screenshot', (req, res) => {});module.exports = router;++module.exports.streamFileWithRange = streamFileWithRange;@@@ -68,6 +68,7 @@ router.post('/:kind/:id', tenantAuth({ allowUser: true }), upload.single('thumbnconst url = `/t/${filename}`;const table = kind === 'vod' ? 'vods' : 'clips';db.run(`UPDATE ${table} SET thumbnail_url = ? WHERE id = ?`, [url, numId]);++ require('../objects/model').safeSync(kind, numId);return res.json({ url });}@@@ -196,6 +196,7 @@ async function generateVodThumbnail(vodId, filePath, opts = {}) {// Clean up the previous thumbnail for this VOD_removeOldThumb(db.get('SELECT thumbnail_url FROM vods WHERE id = ?', [vodId])?.thumbnail_url, thumbUrl);db.run('UPDATE vods SET thumbnail_url = ? WHERE id = ?', [thumbUrl, vodId]);++ require('../objects/model').safeSync('vod', vodId); // the thumbnail object follows the picture}return thumbUrl;}@@@ -206,6 +207,7 @@ async function generateClipThumbnail(clipId, filePath) {if (thumbUrl) {_removeOldThumb(db.get('SELECT thumbnail_url FROM clips WHERE id = ?', [clipId])?.thumbnail_url, thumbUrl);db.run('UPDATE clips SET thumbnail_url = ? WHERE id = ?', [thumbUrl, clipId]);++ require('../objects/model').safeSync('clip', clipId);}return thumbUrl;}@@@ -77,6 +77,7 @@ async function _recutClip(clipId, { reason = 'recut' } = {}) {if (!cut.ok) return fail(clipId, clip, attempt, cut.error);db.run("UPDATE clips SET file_path = ?, duration_seconds = ?, end_time = ?, status = 'ready', cut_error = NULL, cut_next_at = NULL WHERE id = ?",[cut.filePath, cut.duration, startTime + cut.duration, clipId]);++ require('../objects/model').safeSync('clip', clipId);try { await require('../thumbnails/thumbnail-service').generateClipThumbnail(clipId, cut.filePath); } catch { /* */ }console.log(`[Clips] Clip ${clipId} ${reason} OK from vod ${clip.vod_id} (${startTime.toFixed(1)}-${(startTime + cut.duration).toFixed(1)}s, attempt ${attempt})`);sendWebhook(clip.app_id, 'clip.ready', _clipPublic(db.getClipById(clipId))).catch(() => {});@@@ -92,6 +93,7 @@ function fail(clipId, clip, attempt, error) {const mins = BACKOFF_MIN[Math.min(attempt - 1, BACKOFF_MIN.length - 1)];const nextAt = more ? new Date(Date.now() + mins * 60000).toISOString().replace('T', ' ').slice(0, 19) : null;db.run("UPDATE clips SET status = 'failed', cut_error = ?, cut_next_at = ? WHERE id = ?", [msg, nextAt, clipId]);++ require('../objects/model').safeSync('clip', clipId);console.warn(`[Clips] Clip ${clipId} attempt ${attempt}/${MAX_ATTEMPTS} failed: ${msg}${more ? ` — retry in ${mins} min` : ' — giving up'}`);sendWebhook(clip.app_id, 'clip.failed', _clipPublic(db.getClipById(clipId))).catch(() => {});return { ok: false, error: msg, retry_at: nextAt };@@@ -23,6 +23,7 @@ const tools = require('./media-tools');const cutter = require('./clip-cutter');const { tenantAuth, tenantCors } = require('../auth');const { sendWebhook } = require('../webhooks');++const objects = require('../objects/model');const router = express.Router({ mergeParams: true });router.use(tenantCors);@@@ -197,6 +198,7 @@ router.post('/', tenantAuth({ allowUser: true }), clipUpload.single('video'), asif (cut.ok) {db.run('UPDATE clips SET file_path = ?, duration_seconds = ?, end_time = ?, status = ? WHERE id = ?',[cut.filePath, cut.duration, startTime + cut.duration, 'ready', clipId]);++ objects.safeSync('clip', clipId);try {await require('../thumbnails/thumbnail-service').generateClipThumbnail(clipId, cut.filePath);} catch { /* */ }@@@ -207,6 +209,7 @@ router.post('/', tenantAuth({ allowUser: true }), clipUpload.single('video'), as// (attempt 2 pulls a cloud-stored VOD back to local disk first).const nextAt = new Date(Date.now() + 2 * 60000).toISOString().replace('T', ' ').slice(0, 19);db.run("UPDATE clips SET status = 'failed', cut_error = ?, cut_attempts = 1, cut_next_at = ? WHERE id = ?", [String(cut.error || 'cut failed').slice(0, 500), nextAt, clipId]);++ objects.safeSync('clip', clipId);console.warn(`[Clips] Clip ${clipId} failed: ${cut.error} — auto-retry in 2 min`);sendWebhook(appId, 'clip.failed', clipPublic(db.getClipById(clipId))).catch(() => {});}@@@ -351,6 +354,7 @@ router.put('/:id', tenantAuth(), (req, res) => {const t = sanitizeClipTitle(title, '');if (!t) return res.status(400).json({ error: 'Title must be 1-200 characters' });db.run('UPDATE clips SET title = ? WHERE id = ?', [t, clip.id]);++ if (visibility === undefined) objects.safeSync('clip', clip.id);}if (visibility !== undefined) db.setClipVisibility(clip.id, visibility);res.json({ clip: clipPublic(db.getClipById(clip.id, req.appId)) });@@@ -364,6 +368,7 @@ router.delete('/:id', tenantAuth(), (req, res) => {try {const clip = _getClipScoped(req, res);if (!clip) return;++ if (objects.isHeldRow(clip)) return res.status(409).json({ error: 'Clip is under a retention hold', code: 'media.object.held' });// Delete the local file + any offloaded B2/R2 object (clips carry// storage_provider/storage_key like VODs).@@@ -136,6 +136,8 @@ async function finalizeVod(vodId, opts = {}) {return await _doFinalize(vodId, opts);} finally {_finalizing.delete(vodId);++ // Every outcome (ready, quarantined, failed) re-projects the object; a deleted row was already marked by trigger.++ require('../objects/model').safeSync('vod', vodId);}}@@@ -123,6 +123,10 @@ async function _scanPass(deep) {// Delete a VOD's files everywhere + its DB row. Mirrors the manual delete route.function _hardDeleteVod(vod) {try {++ if (require('../objects/model').isHeldRow(db.get('SELECT object_id FROM vods WHERE id = ?', [vod.id]))) {++ console.log(`[VOD-Health] vod ${vod.id} is under a retention hold — not deleting`);++ return false;++ }if (vod.file_path) {try { require('./vod-storage').deleteVodObjects(vod).catch(() => {}); } catch { /* */ }// Local file + sidecars (.seekable.webm / .seekable.mp4, .master.mkv).@@@ -218,6 +218,7 @@ class StreamRecorder {this.activeRecordings.set(vod.id, recording);db.run('UPDATE vods SET is_recording = 1, file_path = ? WHERE id = ?', [filePath, vod.id]);if (masterPath) db.run('UPDATE vods SET master_file_path = ? WHERE id = ?', [masterPath, vod.id]);++ require('../objects/model').safeSync('vod', vod.id);return recording;}@@@ -25,6 +25,7 @@ const recorder = require('./recorder');const tools = require('./media-tools');const { finalizeVod, vodPublic } = require('./finalize');const { tenantAuth, tenantCors } = require('../auth');++const objects = require('../objects/model');const router = express.Router({ mergeParams: true });router.use(tenantCors);@@@ -101,6 +102,7 @@ router.post('/', tenantAuth(), (req, res) => {const id = result.lastInsertRowid;if (clips_only) db.run('UPDATE vods SET clips_only = 1 WHERE id = ?', [id]);if (visibility) db.setVodVisibility(id, visibility);++ else if (!clips_only) objects.safeSync('vod', id);res.status(201).json({ id });} catch (err) {console.error('[VOD] Create error:', err.message);@@@ -342,6 +344,7 @@ router.put('/:id', tenantAuth(), (req, res) => {db.run(`UPDATE vods SET ${updates.join(', ')} WHERE id = ?`, params);}if (visibility !== undefined) db.setVodVisibility(vod.id, visibility);++ else if (updates.length) objects.safeSync('vod', vod.id);res.json({ vod: vodPublic(db.getVodById(vod.id, req.appId)) });} catch (err) {console.error('[VOD] Update error:', err.message);@@@ -354,6 +357,7 @@ router.delete('/:id', tenantAuth(), (req, res) => {try {const vod = _getVodScoped(req, res);if (!vod) return;++ if (objects.isHeldRow(vod)) return res.status(409).json({ error: 'VOD is under a retention hold', code: 'media.object.held' });if (recorder.isRecording(vod.id)) recorder.stopRecording(vod.id);activeChunkUploads.delete(vod.id);@@@ -160,6 +160,16 @@ function clientFor(name) {return clients[name];}++function bucketFor(name) {++ return (PROVIDER_ENV[name] && PROVIDER_ENV[name].bucket) || null;++}++++// Object model (Wave 4): holds freeze an object's placement, and every tier move++// re-projects the VOD's media_locations. Lazy — the model requires this module.++const objects = () => require('../objects/model');++function _held(vod) { return objects().isHeldRow(vod); }++function _moved(vodId, verified) { objects().afterTierMove(vodId, verified); }++function keyForVod(vod) {if (vod.storage_key) return vod.storage_key;return KEY_PREFIX + path.basename(vod.file_path || '');@@@ -374,6 +384,7 @@ async function moveToCold(vodId) {const vod = db.get('SELECT * FROM vods WHERE id = ?', [vodId]);if (!vod || !vod.file_path) return { ok: false, error: 'VOD not found' };if (vod.is_recording) return { ok: false, error: 'VOD is currently recording' };++ if (_held(vod)) return { ok: false, held: true, error: 'VOD is under a retention hold' };if (!providerConfigured('b2')) return { ok: false, error: 'B2 not configured' };const key = keyForVod(vod);@@@ -384,6 +395,7 @@ async function moveToCold(vodId) {// Not local — maybe already offloadedif (await headObject('b2', key)) {db.run("UPDATE vods SET storage_provider = 'b2', storage_key = ? WHERE id = ?", [key, vodId]);++ _moved(vodId, ['b2']);return { ok: true, already: true };}// Neither here nor in B2: nothing to offload, ever. Quarantine the row so the@@@ -403,6 +415,7 @@ async function moveToCold(vodId) {console.error(`[VodStorage] Uploaded but failed to remove local file for VOD ${vodId}:`, err.message);}cleanupSidecar(local);++ _moved(vodId, ['b2']);console.log(`[VodStorage] VOD ${vodId} offloaded to B2: ${key} (${(localSize / 1048576).toFixed(1)} MB)`);return { ok: true, bytes: localSize };@@@ -434,6 +447,7 @@ async function moveToHot(vodId) {if (fs.existsSync(local)) {db.run("UPDATE vods SET storage_provider = 'local' WHERE id = ?", [vodId]);++ _moved(vodId, []);return { ok: true, already: true };}@@@ -483,6 +497,7 @@ async function moveToHot(vodId) {// Restoring to local keeps the B2 canonical copy; drop any R2 copy.if (providerOf(vod) === 'r2' && providerConfigured('r2')) await deleteObject('r2', key);db.run("UPDATE vods SET storage_provider = 'local' WHERE id = ?", [vodId]);++ _moved(vodId, []);console.log(`[VodStorage] VOD ${vodId} restored to local (${(head.size / 1048576).toFixed(1)} MB)`);return { ok: true, bytes: head.size };} catch (err) {@@@ -496,6 +511,7 @@ async function promoteToR2(vodId) {const vod = db.get('SELECT * FROM vods WHERE id = ?', [vodId]);if (!vod || !vod.file_path) return { ok: false, error: 'VOD not found' };if (vod.is_recording) return { ok: false, error: 'VOD is currently recording' };++ if (_held(vod)) return { ok: false, held: true, error: 'VOD is under a retention hold' };if (!providerConfigured('r2') || providerHealthy.r2 === false) return { ok: false, error: 'R2 not available' };if (!providerConfigured('b2')) return { ok: false, error: 'B2 not configured' };@@@ -519,6 +535,7 @@ async function promoteToR2(vodId) {}db.run("UPDATE vods SET storage_provider = 'r2', storage_key = ? WHERE id = ?", [key, vodId]);++ _moved(vodId, ['b2', 'r2']);// Popular VODs live in R2+B2; free the local copylet freed = 0;@@@ -540,6 +557,7 @@ async function demoteFromR2(vodId) {const vod = db.get('SELECT * FROM vods WHERE id = ?', [vodId]);if (!vod) return { ok: false, error: 'VOD not found' };if (providerOf(vod) !== 'r2') return { ok: true, already: true };++ if (_held(vod)) return { ok: false, held: true, error: 'VOD is under a retention hold' };const key = keyForVod(vod);try {@@@ -551,6 +569,7 @@ async function demoteFromR2(vodId) {}await deleteObject('r2', key);db.run("UPDATE vods SET storage_provider = 'b2' WHERE id = ?", [vodId]);++ _moved(vodId, ['b2']);console.log(`[VodStorage] VOD ${vodId} demoted from R2 to B2`);return { ok: true };} catch (err) {@@@ -561,6 +580,8 @@ async function demoteFromR2(vodId) {/** Delete a VOD's (or clip's — same columns) media everywhere (local + B2 + R2). */async function deleteVodObjects(vod) {if (!vod?.file_path) return;++ // Last line of defence for every delete path: a held object keeps its bytes.++ if (_held(vod)) { console.warn(`[VodStorage] Not deleting media of ${vod.object_id}: under a retention hold`); return; }const local = localPathForVod(vod);if (fs.existsSync(local)) {try { fs.unlinkSync(local); } catch { /* ignore */ }@@@ -783,7 +804,7 @@ async function runSweep() {`, [`-${settings.minAgeDays} days`, settings.maxViewsForCold, `-${settings.minLastAccessDays} days`, settings.maxPerSweep]);}-- let skippedBackoff = 0, quarantined = ghosts;++ let skippedBackoff = 0, skippedHeld = 0, quarantined = ghosts;for (const vod of candidates) {if (underPressure && drainSatisfied(diskUsage(config.vod.path), settings)) break;const failedAt = offloadFailedAt.get(vod.id);@@@ -794,6 +815,7 @@ async function runSweep() {const key = keyForVod(vod);if (await headObject('b2', key).catch(() => null)) {db.run("UPDATE vods SET storage_provider = 'b2', storage_key = ? WHERE id = ?", [key, vod.id]);++ _moved(vod.id, ['b2']);} else {quarantineMissing(vod.id, `file not found at ${localPathForVod(vod)}`);quarantined++;@@@ -804,6 +826,8 @@ async function runSweep() {if (result.ok) {offloadFailedAt.delete(vod.id);if (!result.already) { migrated++; bytesFreed += result.bytes || 0; }++ } else if (result.held) {++ skippedHeld++;} else {offloadFailedAt.set(vod.id, Date.now());errors.push({ id: vod.id, error: result.error });@@@ -812,10 +836,11 @@ async function runSweep() {// Say what happened whenever it matters: the old sweep only logged on success, so// weeks of "draining" that freed nothing looked healthy in the journal.-- if (underPressure || errors.length || quarantined) {++ if (underPressure || errors.length || quarantined || skippedHeld) {const freedMb = (bytesFreed / 1048576).toFixed(1);console.log(`[VodStorage] Offload pass: ${candidates.length} candidate(s), ${migrated} uploaded (${freedMb} MB freed), `+ `${errors.length} failed, ${skippedBackoff} in retry back-off, ${quarantined} quarantined`++ + (skippedHeld ? `, ${skippedHeld} under a retention hold` : '')+ (underPressure ? `, disk now ${diskUsage(config.vod.path).usePct}%` : ''));for (const e of errors.slice(0, 5)) console.warn(`[VodStorage] VOD ${e.id}: ${e.error}`);if (underPressure && !migrated && !candidates.length) {@@@ -877,6 +902,7 @@ async function runSweep() {for (const vod of popular) {const result = await promoteToR2(vod.id);if (result.ok && !result.already) { promoted++; bytesFreed += result.bytes || 0; }++ else if (result.held) skippedHeld++;else if (!result.ok) errors.push({ id: vod.id, error: result.error });}@@@ -903,6 +929,7 @@ async function runSweep() {critical,stillNeedsDrain,skippedBackoff,++ skippedHeld,quarantined,diskPct: diskAfter.usePct,freeGb: Number((diskAfter.available / GB).toFixed(1)),@@@ -978,6 +1005,7 @@ async function migrateLegacy() {const head = await headObject('b2', key);if (head) {db.run("UPDATE vods SET storage_provider = 'b2', storage_key = ? WHERE id = ?", [key, vod.id]);++ _moved(vod.id, ['b2']);flipped++;} else if (fs.existsSync(localPathForVod(vod))) {restoredLocal++; // still local, sweep will re-offload@@@ -1212,6 +1240,9 @@ module.exports = {uploadTimeoutMs,providerOf,isRemote,++ providerConfigured,++ bucketFor,++ headObject,keyForVod,localPathForVod,resolvePlayback,nnew file mode 100644@@@ -0,0 +1,270 @@++'use strict';++// Object API v2 (/api/v2/:app/objects) and /o/:id: init → PUT → complete with an app key, an upload++// token or a Network service token; sha256 + size + quota checks; cursor listing; signed private++// downloads; soft delete/restore; retention holds; the public-size invariant at upload time; and the++// legacy projections (read-only through v2; their v1 deletes honour holds).++const assert = require('assert');++const fs = require('fs');++const os = require('os');++const path = require('path');++const http = require('http');++const crypto = require('crypto');++++const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-objapi-'));++const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++process.env.DB_PATH = path.join(tmp, 'media.db');++process.env.VOD_PATH = dir('vods');++process.env.FILES_PATH = dir('files');++process.env.OBJECTS_PATH = dir('objects');++process.env.THUMBNAILS_PATH = dir('thumbnails');++process.env.OV_NETWORK_URL = 'https://openvibe.network';++process.env.MEDIA_PUBLIC_URL = 'https://media.test';++process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';++process.env.MEDIA_OBJECT_MAX_MB = '1';++process.env.MEDIA_PUBLIC_OBJECT_MAX_MB = '0'; // every public playback object violates here++++const db = require('../server/db/database');++const auth = require('../server/auth');++const model = require('../server/objects/model');++const objectRoutes = require('../server/objects/routes');++const { serviceAuth } = require('openvibe-contracts');++const express = require('express');++++const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });++auth._setNetworkPublicKeyForTests(keys.publicKey);++db.upsertApp({ app_id: 'live', api_key: 'live-key' });++db.upsertApp({ app_id: 'games', api_key: 'games-key' });++db.upsertApp({ app_id: 'tiny', api_key: 'tiny-key', quota_bytes: 20 });++const now = Math.floor(Date.now() / 1000);++const tok = (over = {}) => serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub: 'svc:example', actor_type: 'service', aud: ['openvibe.media'], cap: ['media.object.upload', 'media.object.read'], ns: ['live'], iat: now, exp: now + 300, jti: `tok_${crypto.randomBytes(6).toString('hex')}`, ...over }, keys.privateKey);++const SUBJECT = 'usr_01JAB2C3D4E5F6G7H8J9K0MNPQ';++++// Same mount order as server/index.js.++const app = express();++app.put('/api/v2/:app/objects/:id/content', ...objectRoutes.contentHandlers);++app.use(express.json());++app.use('/api/v1/:app/files', require('../server/files/routes'));++app.use('/api/v1/:app/vods', require('../server/vod/routes'));++app.use('/api/v2/:app/objects', objectRoutes);++app.use('/o', objectRoutes.publicRouter);++const server = http.createServer(app);++const sha = (b) => crypto.createHash('sha256').update(b).digest('hex');++++(async () => {++ await new Promise((r) => server.listen(0, '127.0.0.1', r));++ const base = `http://127.0.0.1:${server.address().port}`;++ const local = (u) => base + String(u).replace('https://media.test', '');++ const call = async (method, p, { bearer = 'live-key', body, headers = {}, raw } = {}) => {++ const h = { ...headers };++ if (bearer) h.authorization = `Bearer ${bearer}`;++ let payload;++ if (raw !== undefined) payload = raw;++ else if (body !== undefined) { h['content-type'] = 'application/json'; payload = JSON.stringify(body); }++ const res = await fetch(p.startsWith('http') ? p : base + p, { method, headers: h, body: payload, redirect: 'manual' });++ const text = await res.text();++ let json = null; try { json = JSON.parse(text); } catch { /* not json */ }++ return { status: res.status, body: json, text, headers: res.headers };++ };++ const O = '/api/v2/live/objects';++++ // ── Init → PUT (upload token, no other credential) → complete ──++ const hello = Buffer.from('hello world');++ let r = await call('POST', O, { body: { kind: 'file', mime_type: 'text/plain', size_bytes: hello.length, filename: '../hello.txt', visibility: 'private', content_hash: sha(hello) } });++ assert.strictEqual(r.status, 201, JSON.stringify(r.body));++ const id = r.body.id;++ assert.ok(/^med_[0-9A-HJKMNP-TV-Z]{26}$/.test(id));++ assert.deepStrictEqual([r.body.object.lifecycle_status, r.body.object.metadata.filename, r.body.upload.method, r.body.upload.max_bytes], ['uploading', 'hello.txt', 'PUT', 11]);++ const putUrl = local(r.body.upload.url);++ r = await call('PUT', putUrl, { bearer: null, raw: Buffer.from('hello') });++ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.object.size_mismatch']);++ r = await call('PUT', putUrl.replace(/token=[^&]+/, 'token=1.forged'), { bearer: null, raw: hello });++ assert.deepStrictEqual([r.status, r.body.code], [401, 'media.upload_token.invalid']);++ r = await call('PUT', putUrl, { bearer: null, raw: hello, headers: { 'content-type': 'application/json' } });++ assert.strictEqual(r.status, 200, JSON.stringify(r.body));++ assert.deepStrictEqual([r.body.size_bytes, r.body.content_hash], [11, sha(hello)]);++ const stored = model.objectFilePath(model.getObject(id));++ assert.strictEqual(fs.readFileSync(stored, 'utf8'), 'hello world', 'bytes stored under OBJECTS_PATH/<app>/<id>');++ r = await call('POST', `${O}/${id}/complete`);++ assert.strictEqual(r.status, 200, JSON.stringify(r.body));++ assert.deepStrictEqual([r.body.lifecycle_status, r.body.mime_type, r.body.content_hash, r.body.public_url], ['ready', 'text/plain', sha(hello), null]);++ r = await call('GET', `${O}/${id}`);++ assert.strictEqual(r.status, 200);++ assert.deepStrictEqual(r.body.locations, [{ provider: 'local', storage_class: 'hot', state: 'present', size_bytes: 11, verified_at: r.body.locations[0].verified_at, canonical: true }]);++ assert.ok(!r.text.includes(tmp), 'no server paths in responses');++ console.log('✅ init → PUT (upload token) → complete; size + sha256 checked');++++ // ── Private download: signed, short-lived ──++ r = await call('GET', `/o/${id}`, { bearer: null });++ assert.strictEqual(r.status, 404, 'a private object is not reachable without a signature');++ r = await call('GET', `${O}/${id}/download`);++ assert.strictEqual(r.status, 200);++ assert.strictEqual(r.body.public, false);++ assert.ok(Date.parse(r.body.expires_at) > Date.now());++ const signed = r.body.url;++ r = await call('GET', local(signed), { bearer: null });++ assert.deepStrictEqual([r.status, r.text, r.headers.get('content-type'), r.headers.get('x-content-type-options'), r.headers.get('cache-control')], [200, 'hello world', 'text/plain', 'nosniff', 'private, no-store']);++ assert.strictEqual(r.headers.get('content-disposition'), 'inline; filename="hello.txt"');++ r = await call('GET', local(signed).replace(/sig=[^&]+/, 'sig=AAAA'), { bearer: null });++ assert.strictEqual(r.status, 404, 'tampered signature');++ const past = now - 10;++ const oldSig = crypto.createHmac('sha256', 'test-signing-secret').update(`get\n${id}\n${past}`).digest('base64url');++ r = await call('GET', `/o/${id}?exp=${past}&sig=${oldSig}`, { bearer: null });++ assert.strictEqual(r.status, 404, 'expired signature');++ const putSig = crypto.createHmac('sha256', 'test-signing-secret').update(`put\n${id}\n${now + 100}`).digest('base64url');++ r = await call('GET', `/o/${id}?exp=${now + 100}&sig=${putSig}`, { bearer: null });++ assert.strictEqual(r.status, 404, 'an upload signature is not a download signature');++ r = await call('GET', `${O}/${id}/download?redirect=1`);++ assert.strictEqual(r.status, 302);++ console.log('✅ private objects: signed short-lived URLs only');++++ // ── Public object: app-key PUT, 302 to /o, Range ──++ const png = Buffer.concat([Buffer.from('\x89PNG\r\n\x1a\n', 'binary'), crypto.randomBytes(100)]);++ r = await call('POST', O, { body: { kind: 'screenshot', visibility: 'public' } });++ const pubId = r.body.id;++ r = await call('PUT', `${O}/${pubId}/content`, { raw: png, headers: { 'content-type': 'image/png' } });++ assert.strictEqual(r.status, 200, JSON.stringify(r.body));++ r = await call('POST', `${O}/${pubId}/complete`);++ assert.deepStrictEqual([r.body.mime_type, r.body.public_url], ['image/png', `https://media.test/o/${pubId}`]);++ r = await call('GET', `${O}/${pubId}/download`);++ assert.deepStrictEqual([r.status, r.headers.get('location')], [302, `https://media.test/o/${pubId}`]);++ r = await call('GET', `/o/${pubId}`, { bearer: null, headers: { range: 'bytes=0-7' } });++ assert.deepStrictEqual([r.status, r.headers.get('content-range'), r.headers.get('content-type')], [206, `bytes 0-7/${png.length}`, 'image/png']);++ console.log('✅ public objects: 302 to the public location, ranged bytes at /o/:id');++++ // ── Service tokens ──++ r = await call('POST', O, { bearer: tok(), headers: { 'x-ov-subject': `user:${SUBJECT}` }, body: { kind: 'asset', size_bytes: 3 } });++ assert.strictEqual(r.status, 201, JSON.stringify(r.body));++ assert.deepStrictEqual([r.body.object.owner.subject, r.body.object.metadata.created_by], [SUBJECT, 'svc:example']);++ const svcId = r.body.id;++ r = await call('PUT', `${O}/${svcId}/content`, { bearer: tok(), raw: Buffer.from('abc') });++ assert.strictEqual(r.status, 200);++ r = await call('POST', `${O}/${svcId}/complete`, { bearer: tok() });++ assert.strictEqual(r.body.lifecycle_status, 'ready');++ r = await call('POST', O, { bearer: tok(), headers: { 'x-ov-subject': 'alice' }, body: {} });++ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.object.invalid'], 'owner must be a usr_ subject');++ r = await call('POST', O, { bearer: tok({ ns: ['community'] }), body: {} });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied']);++ r = await call('POST', O, { bearer: tok({ cap: ['media.object.read'] }), body: {} });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'read grant cannot upload');++ r = await call('GET', `${O}?owner=${SUBJECT}`, { bearer: tok({ cap: ['media.object.read'] }) });++ assert.deepStrictEqual(r.body.objects.map(o => o.id), [svcId], 'read grant lists; owner filter');++ r = await call('GET', `${O}/${svcId}`, { bearer: tok({ cap: ['media.object.upload'] }) });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'upload grant cannot read');++ r = await call('POST', `${O}/${svcId}/holds`, { bearer: tok(), body: { kind: 'admin' } });++ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'holds are app-key only');++ r = await call('GET', `/api/v2/games/objects/${svcId}`, { bearer: 'games-key' });++ assert.strictEqual(r.status, 404, 'objects are namespaced');++ r = await call('GET', `/api/v2/games/objects/${svcId}`, { bearer: 'live-key' });++ assert.strictEqual(r.status, 403, "another app's key is refused");++ console.log('✅ service tokens: capability + namespace enforced, X-OV-Subject owns the object');++++ // ── Cursor listing ──++ const page1 = await call('GET', `${O}?limit=2`);++ assert.strictEqual(page1.body.objects.length, 2);++ assert.ok(page1.body.next_cursor);++ const page2 = await call('GET', `${O}?limit=2&cursor=${page1.body.next_cursor}`);++ const seen = [...page1.body.objects, ...page2.body.objects].map(o => o.id);++ assert.deepStrictEqual(seen, [svcId, pubId, id], 'newest first, no duplicates');++ assert.strictEqual(page2.body.next_cursor, null);++ assert.deepStrictEqual((await call('GET', `${O}?kind=screenshot`)).body.objects.map(o => o.id), [pubId]);++ assert.deepStrictEqual((await call('GET', `${O}?visibility=private&kind=file`)).body.objects.map(o => o.id), [id]);++ assert.strictEqual((await call('GET', `${O}?cursor=nope`)).status, 400);++ console.log('✅ cursor listing with filters');++++ // ── Acting users see public + their own ──++ r = await call('POST', O, { headers: { 'x-ov-user-id': '7' }, body: { kind: 'file', visibility: 'private' } });++ const sevens = r.body.id;++ assert.strictEqual(r.body.object.owner.user_id, 7);++ assert.strictEqual((await call('GET', `${O}/${sevens}`, { headers: { 'x-ov-user-id': '8' } })).status, 404);++ assert.ok(!(await call('GET', O, { headers: { 'x-ov-user-id': '8' } })).body.objects.some(o => o.id === sevens || o.id === id), 'no enumeration of others\' private objects');++ assert.ok((await call('GET', O, { headers: { 'x-ov-user-id': '7' } })).body.objects.some(o => o.id === sevens));++ assert.strictEqual((await call('DELETE', `${O}/${pubId}`, { headers: { 'x-ov-user-id': '8' } })).status, 403);++ console.log('✅ acting users: public + own objects only');++++ // ── Holds, soft delete, restore ──++ r = await call('POST', `${O}/${id}/holds`, { body: { kind: 'dmca', reason: 'notice 42' } });++ assert.strictEqual(r.status, 201);++ const holdId = r.body.id;++ assert.strictEqual(r.body.created_by, 'app:live');++ assert.strictEqual((await call('GET', `${O}/${id}`)).body.held, true);++ r = await call('DELETE', `${O}/${id}`);++ assert.deepStrictEqual([r.status, r.body.code], [409, 'media.object.held']);++ assert.strictEqual((await call('POST', `${O}/${id}/holds`, { body: { kind: 'nope' } })).status, 400);++ assert.strictEqual((await call('GET', `${O}/${id}/holds`)).body.holds.length, 1);++ r = await call('DELETE', `${O}/${id}/holds/${holdId}`);++ assert.ok(r.body.released_at);++ assert.strictEqual((await call('GET', `${O}/${id}/holds`)).body.holds.length, 0);++ assert.strictEqual((await call('GET', `${O}/${id}/holds?all=1`)).body.holds.length, 1);++ r = await call('DELETE', `${O}/${id}`);++ assert.deepStrictEqual([r.status, r.body.lifecycle_status], [200, 'deleted']);++ assert.ok(fs.existsSync(stored), 'soft delete keeps the bytes');++ assert.strictEqual((await call('GET', local(signed), { bearer: null })).status, 410, 'a deleted object is gone, even with a valid signature');++ assert.strictEqual((await call('GET', `${O}/${id}/download`)).status, 410);++ assert.ok(!(await call('GET', O)).body.objects.some(o => o.id === id), 'deleted objects leave the default listing');++ assert.ok((await call('GET', `${O}?include_deleted=1`)).body.objects.some(o => o.id === id));++ r = await call('POST', `${O}/${id}/restore`);++ assert.deepStrictEqual([r.status, r.body.lifecycle_status], [200, 'ready']);++ assert.strictEqual((await call('POST', `${O}/${id}/restore`)).status, 409);++ console.log('✅ holds block deletion (409); soft delete keeps bytes; restore');++++ // ── Integrity + limits ──++ r = await call('POST', O, { body: { kind: 'file', content_hash: sha(Buffer.from('expected')) } });++ const badId = r.body.id;++ await call('PUT', `${O}/${badId}/content`, { raw: Buffer.from('something else') });++ r = await call('POST', `${O}/${badId}/complete`);++ assert.deepStrictEqual([r.status, r.body.code], [422, 'media.object.hash_mismatch']);++ assert.strictEqual((await call('POST', O, { body: { kind: 'file', content_hash: 'xyz' } })).status, 400);++ assert.strictEqual((await call('POST', O, { body: { kind: 'nope' } })).status, 400);++ r = await call('POST', O, { body: { kind: 'file', size_bytes: 2 * 1024 * 1024 } });++ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.object.too_large'], 'MEDIA_OBJECT_MAX_MB at init');++ r = await call('POST', O, { body: { kind: 'file' } });++ r = await call('PUT', `${O}/${r.body.id}/content`, { raw: crypto.randomBytes(1024 * 1024 + 10) });++ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.object.too_large'], 'and while streaming');++ r = await call('POST', O, { body: { kind: 'vod', visibility: 'public', size_bytes: 10 } });++ assert.deepStrictEqual([r.status, r.body.code], [422, 'media.invariant.public_object_too_large'], 'public playback objects over the invariant are refused');++ r = await call('POST', O, { body: { kind: 'vod', visibility: 'private', size_bytes: 10 } });++ assert.strictEqual(r.status, 201, 'private ones are not public playback');++ const T = '/api/v2/tiny/objects';++ assert.deepStrictEqual([(r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 30 } })).status, r.body.code], [413, 'media.quota.exceeded']);++ assert.strictEqual((await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 10 } })).status, 201);++ r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 15 } });++ assert.strictEqual(r.status, 413, 'reservations count against the quota at init');++ r = await call('POST', T, { bearer: 'tiny-key', body: {} });++ r = await call('PUT', `${T}/${r.body.id}/content`, { bearer: 'tiny-key', raw: Buffer.alloc(15) });++ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.quota.exceeded'], 'and again with the real size');++ assert.deepStrictEqual(fs.readdirSync(path.join(process.env.OBJECTS_PATH, '.tmp')), [], 'refused uploads leave no temp files');++ console.log('✅ hash mismatch, size limit, invariant and quota enforced');++++ // ── Legacy projections through v2 ──++ r = await call('POST', '/api/v1/live/vods', { body: { title: 'Rec', user_id: 3 } });++ const vodId = r.body.id;++ const legacy = `legacy:live:vod:${vodId}`;++ r = await call('GET', `${O}/${legacy}`);++ assert.deepStrictEqual([r.status, r.body.kind, r.body.lifecycle_status, r.body.legacy_ref], [200, 'vod', 'uploading', legacy], 'a v1 vod creates its object; readable by legacy ref');++ const vodObj = r.body.id;++ assert.strictEqual((await call('DELETE', `${O}/${vodObj}`)).body.code, 'media.object.legacy_managed');++ fs.writeFileSync(path.join(process.env.VOD_PATH, 'v.webm'), 'x');++ db.run('UPDATE vods SET file_path = ?, file_size = 1, duration_seconds = 5 WHERE id = ?', [path.join(process.env.VOD_PATH, 'v.webm'), vodId]);++ model.sync('vod', vodId);++ r = await call('GET', `${O}/${vodObj}/download`);++ assert.deepStrictEqual([r.status, r.headers.get('location')], [302, `https://media.test/v/${vodId}`], 'public legacy objects redirect to their inherited URL');++ await call('POST', `${O}/${vodObj}/holds`, { body: { kind: 'evidence' } });++ r = await call('DELETE', `/api/v1/live/vods/${vodId}`);++ assert.deepStrictEqual([r.status, r.body.code], [409, 'media.object.held'], 'v1 delete honours the hold');++ assert.ok(fs.existsSync(path.join(process.env.VOD_PATH, 'v.webm')));++ const fd = new FormData();++ fd.append('file', new Blob([Buffer.from('data')], { type: 'text/plain' }), 'x.txt');++ r = await fetch(`${base}/api/v1/live/files`, { method: 'POST', headers: { authorization: 'Bearer live-key' }, body: fd }).then(async (x) => ({ status: x.status, body: await x.json() }));++ assert.strictEqual(r.status, 201);++ const fileObj = (await call('GET', `${O}/legacy:live:file:${r.body.key}`)).body;++ assert.deepStrictEqual([fileObj.kind, fileObj.content_hash, fileObj.visibility], ['file', r.body.sha256, 'public'], 'v1 file upload creates its object');++ await call('POST', `${O}/${fileObj.id}/holds`, { body: { kind: 'moderation' } });++ assert.strictEqual((await call('DELETE', `/api/v1/live/files/${r.body.key}`)).status, 409);++ const v1List = await call('GET', '/api/v1/live/files');++ assert.ok(!('object_id' in v1List.body.files[0]), 'v1 responses are unchanged');++ console.log('✅ legacy projections: readable via v2, deleted only via v1, holds honoured there');++++ server.close();++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('object API v2: all checks passed');++})().catch((err) => { console.error(err); server.close(); process.exit(1); });nnew file mode 100644@@@ -0,0 +1,244 @@++'use strict';++// Canonical object model (roadmap Wave 4): the backfill projects every vod, clip, file, screenshot,++// avatar and thumbnail row onto media_objects + media_locations without touching bytes or providers;++// it is idempotent; the write hooks keep the model current; holds block deletion and tier moves on++// every path; the public object-size invariant is recorded as policy.++const assert = require('assert');++const fs = require('fs');++const os = require('os');++const path = require('path');++++const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-objects-'));++const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++process.env.DB_PATH = path.join(tmp, 'media.db');++process.env.VOD_PATH = dir('vods');++process.env.CLIPS_PATH = dir('clips');++process.env.FILES_PATH = dir('files');++process.env.THUMBNAILS_PATH = dir('thumbnails');++process.env.PASTES_PATH = dir('pastes');++process.env.OBJECTS_PATH = dir('objects');++process.env.MEDIA_PUBLIC_URL = 'https://media.test';++process.env.MEDIA_B2_BUCKET = 'bkt-b2'; // bucket name only: no endpoint, so B2 is not configured++process.env.MEDIA_R2_BUCKET = 'bkt-r2';++process.env.MEDIA_PUBLIC_OBJECT_TARGET_MB = '1';++process.env.MEDIA_PUBLIC_OBJECT_WARN_MB = '2';++process.env.MEDIA_PUBLIC_OBJECT_MAX_MB = '3';++++const contracts = require('openvibe-contracts');++const db = require('../server/db/database');++const model = require('../server/objects/model');++const { backfill } = require('../server/objects/backfill');++const invariant = require('../server/objects/invariant');++const vodStorage = require('../server/vod/vod-storage');++++const MB = 1024 * 1024;++const MED = /^med_[0-9A-HJKMNP-TV-Z]{26}$/;++const d = db.getDb();++const write = (p, bytes) => { fs.mkdirSync(path.dirname(p), { recursive: true }); fs.writeFileSync(p, Buffer.alloc(bytes, 7)); return p; };++const objOf = (table, id) => model.getObject(d.prepare(`SELECT object_id FROM ${table} WHERE ${table === 'files' ? 'key' : 'id'} = ?`).get(id).object_id);++const locs = (o) => Object.fromEntries(model.listLocations(o.id).map(l => [l.provider, l]));++++// ── Inherited rows, inserted raw (no hooks), the way the cutover import left them ──++const vodFile = write(path.join(process.env.VOD_PATH, 'vod-live-1.webm'), 1000);++write(path.join(process.env.THUMBNAILS_PATH, 'vod-1-111.jpg'), 50);++const insVod = d.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, file_size, duration_seconds, visibility, is_public, storage_provider, storage_key, thumbnail_url, clips_only, is_recording, created_at)++ VALUES (@id, @app, @user, @title, @file, @size, @dur, @vis, @pub, @prov, @key, @thumb, @co, 0, @at)`);++const V = (o) => insVod.run({ app: 'live', user: 5, title: 'V', file: null, size: 0, dur: 60, vis: 'public', pub: 1, prov: 'local', key: null, thumb: null, co: 0, at: '2025-06-01 10:00:00', ...o });++V({ id: 1, file: vodFile, size: 1000, thumb: '/t/vod-1-111.jpg', at: '2025-01-01 00:00:00' }); // local, present++V({ id: 2, file: '/old/host/vods/old.webm', size: 5 * MB, vis: 'unlisted', pub: 0, prov: 'b2', key: 'vods/old.webm' }); // offloaded++V({ id: 3, file: '/old/host/vods/pop.mp4', size: 2.5 * MB, prov: 'r2', key: 'vods/pop.mp4', thumb: 'https://elsewhere.example/x.jpg' });++V({ id: 4, file: '/opt/hobostreamer/data/vods/gone.webm', size: 50 }); // local but gone++V({ id: 5, file: path.join(process.env.VOD_PATH, 'eph.webm'), co: 1 }); // clips-only++V({ id: 6, dur: 0 }); // ghost: no file yet++V({ id: 7, app: 'games', file: vodFile, size: 1000, vis: 'private', pub: 0 });++const clipFile = write(path.join(process.env.CLIPS_PATH, 'clip-1.webm'), 300);++d.prepare("INSERT INTO clips (id, app_id, vod_id, user_id, title, file_path, status, visibility, is_public, start_time, end_time) VALUES (1, 'live', 1, 9, 'C', ?, 'ready', 'public', 1, 5, 15)").run(clipFile);++d.prepare("INSERT INTO clips (id, app_id, vod_id, user_id, title, file_path, status) VALUES (2, 'live', 1, 9, 'C2', '', 'processing')").run();++write(path.join(process.env.FILES_PATH, 'live', 'abc123-a.txt'), 12);++d.prepare("INSERT INTO files (key, app_id, user_id, original_name, size, mime, sha256) VALUES ('abc123-a.txt', 'live', 3, 'a.txt', 12, 'text/plain', ?)").run('a'.repeat(64));++const shot = write(path.join(process.env.PASTES_PATH, 'screenshots', 's1.png'), 40);++const avatar = write(path.join(process.env.PASTES_PATH, 'screenshots', 'avatar-n4.webp'), 30);++d.prepare("INSERT INTO pastes (id, app_id, slug, user_id, type, title, content, visibility, screenshot_path, metadata) VALUES (1, 'live', 'shot-one', 2, 'screenshot', 'S', '', 'public', ?, NULL)").run(shot);++d.prepare("INSERT INTO pastes (id, app_id, slug, user_id, type, title, content, visibility, screenshot_path, metadata) VALUES (2, 'network', 'avatar-four', 4, 'screenshot', 'Avatar', '', 'unlisted', ?, ?)").run(avatar, JSON.stringify({ kind: 'avatar', mime_type: 'image/webp' }));++d.prepare("INSERT INTO pastes (id, app_id, slug, type, title, content, visibility) VALUES (3, 'live', 'text-three', 'paste', 'T', 'hello', 'public')").run();++d.prepare("INSERT INTO pastes (id, app_id, slug, type, title, content, visibility, screenshot_path) VALUES (4, 'live', 'shot-nopath', 'screenshot', 'S', '', 'public', NULL)").run();++const mtimes = [vodFile, clipFile, shot, avatar].map(p => fs.statSync(p).mtimeMs);++++// ── Dry run: a full report, nothing written ──++const dry = backfill({ dryRun: true });++assert.strictEqual(dry.dry_run, true);++assert.deepStrictEqual(dry.counts.vod, { seen: 7, created: 6, updated: 0, skipped: 1 });++assert.strictEqual(d.prepare('SELECT COUNT(*) c FROM media_objects').get().c, 0, 'dry run writes no objects');++assert.strictEqual(d.prepare('SELECT COUNT(*) c FROM vods WHERE object_id IS NOT NULL').get().c, 0, 'dry run links no rows');++assert.strictEqual(model.parseJson(d.prepare("SELECT value FROM media_settings WHERE key = 'objects.backfill.last_report'").get()?.value, null), null, 'dry run stores no report');++console.log('✅ dry run reports without writing');++++// ── Real run ──++const r = backfill();++assert.deepStrictEqual(r.counts.vod, { seen: 7, created: 6, updated: 0, skipped: 1 });++assert.deepStrictEqual(r.counts.clip, { seen: 2, created: 2, updated: 0, skipped: 0 });++assert.deepStrictEqual(r.counts.file, { seen: 1, created: 1, updated: 0, skipped: 0 });++assert.deepStrictEqual(r.counts.screenshot, { seen: 2, created: 1, updated: 0, skipped: 1 });++assert.deepStrictEqual(r.counts.avatar, { seen: 1, created: 1, updated: 0, skipped: 0 });++assert.deepStrictEqual(r.counts.thumbnail, { seen: 2, created: 1, updated: 0, skipped: 1 });++assert.deepStrictEqual(r.skipped.map(s => s.reason).sort(), ['clips-only recording (ephemeral, never published)', 'external thumbnail url', 'screenshot paste without a file path']);++assert.deepStrictEqual(r.errors, []);++assert.ok(d.prepare("SELECT value FROM media_settings WHERE key = 'objects.backfill.last_report'").get(), 'report recorded');++assert.deepStrictEqual([vodFile, clipFile, shot, avatar].map(p => fs.statSync(p).mtimeMs), mtimes, 'no bytes touched');++++const v1 = objOf('vods', 1), v2 = objOf('vods', 2), v3 = objOf('vods', 3), v4 = objOf('vods', 4), v6 = objOf('vods', 6), v7 = objOf('vods', 7);++for (const o of [v1, v2, v3, v4, v6, v7]) {++ assert.ok(MED.test(o.id), o.id);++ assert.ok(contracts.validate('media.media-ref@1', { media_id: o.id }).valid, 'canonical id is a valid media-ref');++ assert.ok(contracts.validate('media.media-ref@1', { media_id: o.legacy_ref }).valid, `legacy ref valid: ${o.legacy_ref}`);++}++assert.ok(v1.id < v2.id, 'ids sort by the original creation time');++assert.strictEqual(v1.legacy_ref, 'legacy:live:vod:1');++assert.deepStrictEqual([v1.kind, v1.visibility, v1.lifecycle_status, v1.size_bytes, v1.owner_app, v1.owner_user_id, v1.mime_type], ['vod', 'public', 'ready', 1000, 'live', 5, 'video/webm']);++assert.strictEqual(locs(v1).local.state, 'present');++assert.strictEqual(locs(v1).local.key, vodFile);++assert.ok(locs(v1).local.verified_at, 'a local file that exists is verified present');++assert.deepStrictEqual([v1.canonical_provider, v1.canonical_key], ['local', vodFile]);++assert.deepStrictEqual(Object.keys(locs(v2)), ['b2'], 'offloaded vod: B2 copy only');++assert.deepStrictEqual([locs(v2).b2.state, locs(v2).b2.bucket, locs(v2).b2.key, locs(v2).b2.storage_class, v2.canonical_provider], ['pending', 'bkt-b2', 'vods/old.webm', 'cold', 'b2']);++assert.deepStrictEqual(Object.keys(locs(v3)).sort(), ['b2', 'r2'], 'R2 cache implies the B2 canonical');++assert.deepStrictEqual([locs(v3).r2.state, locs(v3).r2.storage_class, v3.canonical_provider], ['pending', 'cache', 'b2']);++assert.strictEqual(locs(v4).local.state, 'missing', 'a local file that is not on disk is recorded missing');++assert.strictEqual(model.listLocations(v6.id).length, 0);++assert.strictEqual(v6.lifecycle_status, 'uploading');++assert.deepStrictEqual([v7.app_id, v7.namespace, v7.visibility], ['games', 'games', 'private']);++assert.strictEqual(d.prepare('SELECT object_id FROM vods WHERE id = 5').get().object_id, null, 'clips-only row stays unprojected');++++const c1 = objOf('clips', 1), c2 = objOf('clips', 2);++assert.deepStrictEqual([c1.kind, c1.lifecycle_status, c1.size_bytes, locs(c1).local.state], ['clip', 'ready', 300, 'present']);++assert.strictEqual(c2.lifecycle_status, 'uploading', 'a clip still being cut is uploading');++const rel = d.prepare("SELECT * FROM media_relationships WHERE from_object_id = ? AND relation = 'clip_of'").get(c1.id);++assert.strictEqual(rel.to_object_id, v1.id, 'clip_of links the clip to its vod');++assert.deepStrictEqual(JSON.parse(rel.metadata), { start_time: 5, end_time: 15 });++++const f = objOf('files', 'abc123-a.txt');++assert.deepStrictEqual([f.kind, f.visibility, f.content_hash, f.mime_type, f.legacy_ref, locs(f).local.checksum], ['file', 'public', 'a'.repeat(64), 'text/plain', 'legacy:live:file:abc123-a.txt', 'a'.repeat(64)]);++const s1 = objOf('pastes', 1), av = objOf('pastes', 2);++assert.deepStrictEqual([s1.kind, s1.legacy_ref, s1.size_bytes], ['screenshot', 'legacy:live:paste:shot-one', 40]);++assert.deepStrictEqual([av.kind, av.app_id, av.legacy_ref, av.visibility, av.mime_type], ['avatar', 'network', 'legacy:network:avatar:avatar-four', 'unlisted', 'image/webp']);++assert.strictEqual(d.prepare('SELECT object_id FROM pastes WHERE id = 3').get().object_id, null, 'text pastes have no bytes object');++++const tv = model.getVariant(v1.id, 'thumbnail');++const th = model.getObject(tv.derived_object_id);++assert.deepStrictEqual([th.kind, th.legacy_ref, th.visibility, locs(th).local.state], ['thumbnail', 'legacy:live:thumbnail:vod-1-111.jpg', 'public', 'present']);++assert.ok(d.prepare("SELECT 1 FROM media_relationships WHERE from_object_id = ? AND relation = 'thumbnail_of' AND to_object_id = ?").get(th.id, v1.id));++console.log('✅ backfill: one object per row, locations mirror where the bytes are, relationships + thumbnail variants');++++// ── Idempotent ──++const countAll = () => ['media_objects', 'media_locations', 'media_relationships', 'media_variants'].map(t => d.prepare(`SELECT COUNT(*) c FROM ${t}`).get().c);++const before = countAll();++const again = backfill();++assert.strictEqual(again.totals.created, 0, 'a re-run creates nothing');++assert.strictEqual(again.counts.vod.updated, 6);++assert.deepStrictEqual(countAll(), before);++assert.strictEqual(objOf('vods', 1).id, v1.id, 'ids are stable');++assert.strictEqual(backfill({ onlyMissing: true }).totals.created, 0, 'only-missing finds nothing new');++console.log('✅ backfill is idempotent');++++// ── Remote knowledge survives a re-projection ──++model.setLocationState(locs(v2).b2.id, { state: 'present', size_bytes: 5 * MB });++model.sync('vod', 2);++assert.strictEqual(locs(v2).b2.state, 'present', 'a verified remote copy is not downgraded to pending');++console.log('✅ verified remote state is kept across syncs');++++// ── Write hooks on the inherited APIs ──++write(path.join(process.env.FILES_PATH, 'live', 'def456-b.png'), 20);++db.createFile({ key: 'def456-b.png', app_id: 'live', user_id: 3, original_name: 'b.png', size: 20, mime: 'image/png', sha256: 'b'.repeat(64) });++assert.strictEqual(objOf('files', 'def456-b.png').content_hash, 'b'.repeat(64), 'file upload creates its object');++db.setVodVisibility(1, 'private');++assert.strictEqual(model.getObject(v1.id).visibility, 'private', 'visibility follows');++assert.strictEqual(model.getObject(th.id).visibility, 'unlisted', 'a private vod thumbnail is still reachable by name: unlisted');++// Thumbnail regenerated: same object, new file.++write(path.join(process.env.THUMBNAILS_PATH, 'vod-1-222.jpg'), 60);++db.run("UPDATE vods SET thumbnail_url = '/t/vod-1-222.jpg' WHERE id = 1");++model.sync('vod', 1);++const th2 = model.getObject(model.getVariant(v1.id, 'thumbnail').derived_object_id);++assert.strictEqual(th2.id, th.id, 'one thumbnail object per parent, updated in place');++assert.deepStrictEqual([th2.legacy_ref, th2.size_bytes], ['legacy:live:thumbnail:vod-1-222.jpg', 60]);++const newClip = db.createClip({ app_id: 'live', vod_id: 1, user_id: 9, title: 'N', file_path: clipFile, status: 'ready' }).lastInsertRowid;++assert.strictEqual(objOf('clips', newClip).kind, 'clip', 'clip creation creates its object');++db.run('DELETE FROM clips WHERE id = ?', [newClip]);++const deadClip = d.prepare('SELECT * FROM media_objects WHERE legacy_ref = ?').get(`legacy:live:clip:${newClip}`);++assert.strictEqual(deadClip.lifecycle_status, 'deleted', 'deleting the row marks its object deleted (trigger — every delete path)');++assert.ok(deadClip.deleted_at);++console.log('✅ inherited writes keep the model current');++++// ── Tier move re-projection ──++const v4row = d.prepare('SELECT * FROM vods WHERE id = 4').get();++db.run("UPDATE vods SET storage_provider = 'b2', storage_key = 'vods/gone.webm' WHERE id = 4");++model.afterTierMove(4, ['b2']);++assert.deepStrictEqual(Object.keys(locs(v4)), ['b2'], 'after offload: the local row goes, the B2 copy appears');++assert.strictEqual(locs(v4).b2.state, 'present', 'the copy the move verified is present');++assert.ok(locs(v4).b2.verified_at);++assert.ok(v4row);++console.log('✅ tier moves update media_locations');++++(async () => {++ // ── Retention holds ──++ const hold = model.placeHold({ object_id: v1.id, kind: 'dmca', reason: 'notice #1', created_by: 'admin:alex' });++ assert.strictEqual(model.isHeld(v1.id), true);++ assert.throws(() => db.run('DELETE FROM vods WHERE id = 1'), /retention hold/, 'a held row cannot be deleted by any path');++ assert.ok(d.prepare('SELECT 1 FROM vods WHERE id = 1').get(), 'row still there');++ assert.throws(() => db.run("UPDATE media_objects SET lifecycle_status = 'deleted' WHERE id = ?", [v1.id]), /retention hold/);++ await vodStorage.deleteVodObjects(d.prepare('SELECT * FROM vods WHERE id = 1').get());++ assert.ok(fs.existsSync(vodFile), 'held bytes are not deleted');++ const cold = await vodStorage.moveToCold(1);++ assert.deepStrictEqual([cold.ok, cold.held], [false, true], 'held objects are not demoted to B2');++ const prom = await vodStorage.promoteToR2(1);++ assert.strictEqual(prom.held, true, 'nor promoted');++ db.run("UPDATE vods SET storage_provider = 'r2' WHERE id = 1");++ assert.strictEqual((await vodStorage.demoteFromR2(1)).held, true, 'nor demoted from R2');++ db.run("UPDATE vods SET storage_provider = 'local' WHERE id = 1");++ assert.throws(() => model.placeHold({ object_id: v1.id, kind: 'whim' }), /hold kind/);++ model.releaseHold(hold.id, 'admin:alex');++ assert.strictEqual(model.isHeld(v1.id), false);++ assert.strictEqual(model.listHolds(v1.id, { includeReleased: true })[0].released_by, 'admin:alex');++ console.log('✅ holds block deletion and tier moves until released');++++ // ── Invariant (thresholds from env: target 1, warn 2, max 3 MB) ──++ assert.strictEqual(invariant.classify(0.5 * MB), 'ok');++ assert.strictEqual(invariant.classify(1.5 * MB), 'above_target');++ assert.strictEqual(invariant.classify(2.5 * MB), 'warn');++ assert.strictEqual(invariant.classify(5 * MB), 'violation');++ assert.strictEqual(invariant.wouldViolate({ kind: 'vod', visibility: 'private', size_bytes: 9 * MB }), false, 'private objects are not public playback');++ assert.strictEqual(invariant.wouldViolate({ kind: 'file', visibility: 'public', size_bytes: 9 * MB }), false, 'only playback kinds');++ const rows = () => d.prepare('SELECT o.legacy_ref, v.level, v.resolved_at FROM media_invariant_violations v JOIN media_objects o ON o.id = v.object_id ORDER BY o.legacy_ref').all();++ assert.deepStrictEqual(rows().map(x => [x.legacy_ref, x.level, x.resolved_at]), [['legacy:live:vod:2', 'violation', null], ['legacy:live:vod:3', 'warn', null]], 'sync records public playback objects over warn/max');++ const scan = invariant.scan({ record: false });++ assert.deepStrictEqual(scan.counts, { ok: 2, above_target: 0, warn: 1, violation: 1 }); // ok: clip 1, vod 4 (vod 1 is private now)++ db.setVodVisibility(3, 'private');++ assert.ok(rows().find(x => x.legacy_ref === 'legacy:live:vod:3').resolved_at, 'making it private resolves the row');++ console.log('✅ public object-size invariant is policy: classified, recorded, resolved');++++ // ── Native soft delete + purge ──++ const nid = model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: 'ready', visibility: 'private', size_bytes: 4 });++ const npath = write(model.objectFilePath(model.getObject(nid)), 4);++ model.upsertLocation(nid, { provider: 'local', key: npath, state: 'present', size_bytes: 4 });++ const heldId = model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: 'ready', size_bytes: 4 });++ const hpath = write(model.objectFilePath(model.getObject(heldId)), 4);++ model.upsertLocation(heldId, { provider: 'local', key: hpath, state: 'present' });++ const deleted = model.softDelete(model.getObject(nid), { by: 'test' });++ assert.strictEqual(deleted.lifecycle_status, 'deleted');++ assert.ok(model.parseJson(deleted.metadata).retention_until, 'retention recorded');++ assert.ok(fs.existsSync(npath), 'soft delete keeps the bytes');++ assert.strictEqual(model.purgeExpired({ retentionDays: 1 }), 0, 'inside the retention period nothing is purged');++ const h2 = model.placeHold({ object_id: heldId, kind: 'evidence' });++ assert.throws(() => model.softDelete(model.getObject(heldId)), (e) => e.code === 'media.object.held');++ model.releaseHold(h2.id);++ model.softDelete(model.getObject(heldId));++ model.placeHold({ object_id: heldId, kind: 'evidence' }); // held after deletion: still kept++ assert.strictEqual(model.purgeExpired({ retentionDays: 0 }), 1);++ assert.ok(!fs.existsSync(npath), 'expired native bytes are purged');++ assert.ok(fs.existsSync(hpath), 'held bytes survive the purge');++ assert.ok(model.parseJson(model.getObject(nid).metadata).purged_at);++ assert.strictEqual(model.restore(model.getObject(nid)), null, 'a purged object cannot be restored');++ console.log('✅ native soft delete keeps bytes for the retention period; purge skips held objects');++++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('objects model + backfill: all checks passed');++})().catch((err) => { console.error(err); process.exit(1); });nnew file mode 100644@@@ -0,0 +1,142 @@++'use strict';++// Reconciliation of the object model against the bytes, with a fake B2/R2 provider: read-only by++// default, --verify records what the HEADs found; it detects missing canonical copies, lost local++// files, size/hash mismatches, orphan locations, deleted-but-still-served objects and unprojected++// rows. Plus the three scripts end to end (backfill --dry-run, reconcile, invariant).++const assert = require('assert');++const fs = require('fs');++const os = require('os');++const path = require('path');++const crypto = require('crypto');++const { execFileSync } = require('child_process');++++const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-reconcile-'));++const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };++const env = {++ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),++ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'), MEDIA_PUBLIC_URL: 'https://media.test',++ MEDIA_PUBLIC_OBJECT_TARGET_MB: '1', MEDIA_PUBLIC_OBJECT_WARN_MB: '2', MEDIA_PUBLIC_OBJECT_MAX_MB: '3',++};++Object.assign(process.env, env);++++const db = require('../server/db/database');++const model = require('../server/objects/model');++const { reconcile, summarize } = require('../server/objects/reconcile');++++const MB = 1024 * 1024;++const d = db.getDb();++const write = (p, content) => { fs.mkdirSync(path.dirname(p), { recursive: true }); fs.writeFileSync(p, content); return p; };++const V = d.prepare(`INSERT INTO vods (id, app_id, file_path, file_size, duration_seconds, visibility, is_public, storage_provider, storage_key)++ VALUES (?, 'live', ?, ?, 60, 'public', 1, ?, ?)`);++++// Rows: a healthy local vod; an offloaded one whose B2 copy is fine; one whose B2 copy is gone but++// R2 still has it; one whose B2 copy has the wrong size; a local vod whose file vanished.++V.run(1, write(path.join(env.VOD_PATH, 'a.webm'), Buffer.alloc(100)), 100, 'local', null);++V.run(2, '/gone/b.webm', 200, 'b2', 'vods/b.webm');++V.run(3, '/gone/c.webm', 300, 'r2', 'vods/c.webm');++V.run(4, '/gone/d.webm', 400, 'b2', 'vods/d.webm');++V.run(5, write(path.join(env.VOD_PATH, 'e.webm'), Buffer.alloc(10)), 10, 'local', null);++const fileBytes = Buffer.from('file-bytes');++write(path.join(env.FILES_PATH, 'live', 'k1-f.bin'), fileBytes);++d.prepare("INSERT INTO files (key, app_id, size, mime, sha256) VALUES ('k1-f.bin', 'live', ?, 'application/octet-stream', ?)").run(fileBytes.length, crypto.createHash('sha256').update(fileBytes).digest('hex'));++++// Scripts: a dry-run backfill reports and writes nothing.++const run = (script, args = []) => {++ try { return { code: 0, out: execFileSync(process.execPath, [path.join(__dirname, '..', 'scripts', script), ...args], { env: { ...process.env, ...env }, encoding: 'utf8' }) }; }++ catch (e) { return { code: e.status, out: String(e.stdout) }; }++};++const dry = JSON.parse(run('backfill-objects.js', ['--dry-run', '--json']).out);++assert.deepStrictEqual([dry.dry_run, dry.counts.vod.created, dry.counts.file.created], [true, 5, 1]);++assert.strictEqual(d.prepare('SELECT COUNT(*) c FROM media_objects').get().c, 0, 'dry-run script wrote nothing');++const real = run('backfill-objects.js');++assert.strictEqual(real.code, 0, real.out);++assert.match(real.out, /vod 5\+\/0~\/0 skipped/);++assert.strictEqual(d.prepare('SELECT COUNT(*) c FROM media_objects').get().c, 6);++console.log('✅ backfill script: --dry-run reports only; a real run projects');++++fs.unlinkSync(path.join(env.VOD_PATH, 'e.webm')); // vod 5 loses its file after backfill++const objId = (vodId) => d.prepare('SELECT object_id FROM vods WHERE id = ?').get(vodId).object_id;++// Anomalies the reconciler must notice:++d.prepare("INSERT INTO media_locations (object_id, provider, key, state) VALUES ('med_01JAB2C3D4E5F6G7H8J9K0MNPQ', 'local', '/nowhere', 'present')").run(); // orphan location++db.run("UPDATE media_objects SET size_bytes = 999 WHERE id = ?", [objId(1)]); // local size mismatch++db.run("UPDATE media_objects SET content_hash = ? WHERE legacy_ref = 'legacy:live:file:k1-f.bin'", ['0'.repeat(64)]); // hash mismatch++db.run("UPDATE media_objects SET lifecycle_status = 'deleted' WHERE id = ?", [objId(2)]); // deleted, but /v/2 still serves it++d.prepare("INSERT INTO clips (id, app_id, file_path, status) VALUES (9, 'live', '/x.webm', 'ready')").run(); // row with no object++const ready = model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: 'ready' }); // ready object, no location++++// Fake provider: key → size (absent = 404). B2 lost c.webm (R2 still has it) and holds a wrong-size d.webm.++const bucket = { b2: { 'vods/b.webm': 200, 'vods/d.webm': 12345 }, r2: { 'vods/c.webm': 300 } };++const heads = [];++const fakeHead = async (provider, key) => { heads.push(`${provider}:${key}`); const s = bucket[provider][key]; return s === undefined ? null : { size: s, etag: 'x' }; };++++(async () => {++ const snapshot = () => JSON.stringify(d.prepare('SELECT * FROM media_locations ORDER BY id').all());++ const before = snapshot();++ const ro = await reconcile({ head: fakeHead, hash: true });++ assert.strictEqual(snapshot(), before, 'default mode writes nothing');++ assert.strictEqual(heads.length, 0, 'default mode calls no provider');++ const count = (rep, k) => rep.issues[k].count;++ assert.strictEqual(count(ro, 'orphan_location'), 1);++ assert.strictEqual(count(ro, 'missing_local_file'), 1);++ assert.strictEqual(ro.issues.missing_local_file.items[0].legacy_ref, 'legacy:live:vod:5');++ assert.strictEqual(count(ro, 'size_mismatch'), 1, 'local size mismatch');++ assert.strictEqual(count(ro, 'hash_mismatch'), 1);++ assert.strictEqual(count(ro, 'deleted_publicly_reachable'), 1);++ assert.strictEqual(ro.issues.deleted_publicly_reachable.items[0].table, 'vods');++ assert.strictEqual(count(ro, 'missing_projection'), 1);++ assert.deepStrictEqual(ro.issues.missing_projection.items[0], { table: 'clips', id: 9 });++ assert.strictEqual(count(ro, 'no_canonical_location'), 1);++ assert.strictEqual(ro.issues.no_canonical_location.items[0].object_id, ready);++ assert.strictEqual(count(ro, 'canonical_missing_replica_present'), 0, 'remote copies are unverified until --verify');++ assert.strictEqual(ro.counts.remote_unverified, 4, 'b2 ×3 + r2 ×1 pending');++ assert.ok(summarize(ro).includes('orphan_location: 1'));++ console.log('✅ read-only pass: local checks, orphans, projections, deleted-but-served — no writes, no provider calls');++++ const v = await reconcile({ verify: true, head: fakeHead });++ assert.strictEqual(heads.length, 4, 'every remote location HEADed once');++ assert.strictEqual(count(v, 'remote_missing'), 1, 'vod 3 canonical B2 copy is gone');++ assert.strictEqual(v.issues.remote_missing.items[0].key, 'vods/c.webm');++ assert.strictEqual(count(v, 'canonical_missing_replica_present'), 1);++ assert.deepStrictEqual(v.issues.canonical_missing_replica_present.items[0].present, ['r2'], 'the R2 cache still has it');++ assert.strictEqual(count(v, 'size_mismatch'), 2, 'local vod 1 + remote vod 4');++ assert.strictEqual(count(v, 'no_present_copy'), 2, 'vod 4 (only copy corrupt) and vod 5 (only copy missing)');++ const st = (vodId, p) => d.prepare('SELECT state, verified_at, size_bytes FROM media_locations WHERE object_id = ? AND provider = ?').get(objId(vodId), p);++ assert.strictEqual(st(3, 'b2').state, 'missing');++ assert.strictEqual(st(3, 'r2').state, 'present');++ assert.strictEqual(st(4, 'b2').state, 'corrupt');++ assert.strictEqual(st(4, 'b2').size_bytes, 12345);++ assert.strictEqual(st(5, 'local').state, 'missing');++ assert.ok(st(3, 'r2').verified_at, '--verify records verified_at');++ assert.ok(v.counts.locations_updated >= 6);++ console.log('✅ --verify: HEADs remotes (fake provider), records states; detects lost canonical with a surviving replica');++++ // An unconfigured provider is "unverifiable", never "missing".++ const u = await reconcile({ verify: true, head: async () => undefined });++ assert.strictEqual(u.counts.remote_unverifiable, 4);++ assert.strictEqual(count(u, 'remote_missing'), 0);++++ // The scripts against the same database.++ const rc = run('reconcile-objects.js', ['--json']);++ assert.strictEqual(rc.code, 1, 'issues → exit 1');++ assert.ok(JSON.parse(rc.out).issues.orphan_location.count === 1);++ // A deleted object's thumbnail that is still served is reachable too.++ write(path.join(env.THUMBNAILS_PATH, 'vod-1-5.jpg'), 'jpg');++ db.run("UPDATE vods SET thumbnail_url = '/t/vod-1-5.jpg' WHERE id = 1");++ model.sync('vod', 1);++ db.run('DELETE FROM vods WHERE id = 1');++ const t = await reconcile({});++ assert.ok(t.issues.deleted_publicly_reachable.items.some(i => i.reason.startsWith('derivative')), 'thumbnail of a deleted vod is flagged');++++ db.run('UPDATE vods SET file_size = ? WHERE id = 3', [5 * MB]);++ model.sync('vod', 3);++ const inv = run('object-invariant.js', ['--json']);++ const invReport = JSON.parse(inv.out);++ assert.strictEqual(invReport.counts.violation, 1);++ assert.strictEqual(invReport.objects[0].legacy_ref, 'legacy:live:vod:3');++ assert.strictEqual(d.prepare("SELECT level FROM media_invariant_violations WHERE object_id = ? AND resolved_at IS NULL").get(objId(3)).level, 'violation');++ assert.match(run('object-invariant.js', ['--dry-run']).out, /violation 1/);++ console.log('✅ scripts: reconcile exit codes + JSON, invariant report records violations');++++ fs.rmSync(tmp, { recursive: true, force: true });++ console.log('objects reconciliation: all checks passed');++})().catch((err) => { console.error(err); process.exit(1); });
@@ -23,7 +23,8 @@ server/
@@ -23,7 +23,8 @@ server/
config.js env config (PORT, DB_PATH, *_PATH, MEDIA_B2_*/R2_*, RTP pool, OV_NETWORK_URL)
config.js env config (PORT, DB_PATH, *_PATH, MEDIA_B2_*/R2_*, RTP pool, OV_NETWORK_URL)
auth.js tenancy + auth middleware, JWKS fetch, app seeding
auth.js tenancy + auth middleware, JWKS fetch, app seeding
webhooks.js HMAC-signed outbound webhooks
webhooks.js HMAC-signed outbound webhooks
- db/schema.sql apps, vods, clips, pastes(+likes/comments), files, content_views, media_settings
+ db/schema.sql apps, vods, clips, pastes(+likes/comments), files, content_views, media_settings,
+ media_objects/locations/relationships/variants/jobs/holds/invariant_violations
db/database.js better-sqlite3 helpers, all app_id-scoped; legacy row importer
db/database.js better-sqlite3 helpers, all app_id-scoped; legacy row importer
vod/recorder.js ffmpeg recording: RTMP pull + RTP (SDP) ingest, codec passthrough
vod/recorder.js ffmpeg recording: RTMP pull + RTP (SDP) ingest, codec passthrough
vod/media-tools.js probes, seekable remux, DVR sidecar, chunk-segment concat
vod/media-tools.js probes, seekable remux, DVR sidecar, chunk-segment concat
@@ -39,8 +40,11 @@ server/
@@ -39,8 +40,11 @@ server/
admin/routes.js /api/v1/:app/admin/storage (disk, tiers, buckets, bulk ops)
admin/routes.js /api/v1/:app/admin/storage (disk, tiers, buckets, bulk ops)
thumbnails/ thumbnail service + /api/v1/:app/thumbnails
thumbnails/ thumbnail service + /api/v1/:app/thumbnails
public/routes.js public /v /c /p /t /f
public/routes.js public /v /c /p /t /f
+ objects/ canonical object model: model (projections, holds), routes (/api/v2 + /o),
+ backfill, reconcile, invariant, signing — see docs/object-model.md
vendor/openvibe-shared/ vendored shared helpers (do not edit; re-sync from canonical)
vendor/openvibe-shared/ vendored shared helpers (do not edit; re-sync from canonical)
scripts/smoke-test.sh end-to-end smoke test (boots a temp instance)
scripts/smoke-test.sh end-to-end smoke test (boots a temp instance)
+scripts/backfill-objects.js / reconcile-objects.js / object-invariant.js object-model operator tools
```
```
## Visitor sign-in
## Visitor sign-in
@@ -192,7 +196,7 @@ directories are shared across apps) and responses carry a `note` saying so.
@@ -192,7 +196,7 @@ directories are shared across apps) and responses carry a `note` saying so.
### Webhooks (outbound)
### Webhooks (outbound)
`POST` to the app's `webhook_url` with body
`POST` to the app's `webhook_url` with body
-`{ "event": "vod.ready"|"vod.failed"|"clip.ready"|"clip.failed", "app_id", "data" }`
+`{ "event": "vod.ready"|"vod.failed"|"clip.ready"|"clip.failed"|"media.object.uploaded", "app_id", "data" }`
and header `X-OVMedia-Signature: sha256=<hex hmac-sha256 of the raw body with
and header `X-OVMedia-Signature: sha256=<hex hmac-sha256 of the raw body with
the app's webhook_secret>`. 3 attempts with backoff, 10 s timeout.
the app's webhook_secret>`. 3 attempts with backoff, 10 s timeout.
@@ -216,6 +220,42 @@ the app's webhook_secret>`. 3 attempts with backoff, 10 s timeout.
@@ -216,6 +220,42 @@ the app's webhook_secret>`. 3 attempts with backoff, 10 s timeout.
Private items respond 403/404 unless the request bears the owning app's API
Private items respond 403/404 unless the request bears the owning app's API
key or the owning user's JWT.
key or the owning user's JWT.
+`GET /o/:id` serves object bytes by canonical id (`med_…`): public/unlisted
+objects openly, private ones only with a valid signature from
+`/api/v2/:app/objects/:id/download`, deleted ones 410 (see below).
+
+## Object model and API v2
+
+Every stored blob is a **media object** (`med_<ULID>`, `media_objects`) with one
+`media_locations` row per copy (local / B2 canonical / R2 cache, each `present`,
+`missing`, `pending` or `corrupt`). The `vods`, `clips`, `files` and screenshot
+`pastes` rows are typed projections over objects (`object_id` column); every
+existing route and response is unchanged, and the old write paths keep the
+model current. Full reference: **[docs/object-model.md](docs/object-model.md)**.
+
+- **Backfill** — `node scripts/backfill-objects.js [--dry-run]`: one object per
+ vod, clip, file, screenshot, avatar and thumbnail; idempotent; never moves
+ bytes or calls B2/R2 (remote copies stay `pending`). The service runs the
+ `--only-missing` form 15 s after boot.
+- **API** — `/api/v2/:app/objects`: init → `PUT /:id/content` (sha256, size,
+ quota) → `/:id/complete`; `GET /:id`, cursor `GET /`, soft `DELETE /:id`
+ (bytes kept `MEDIA_DELETE_RETENTION_DAYS`), `/:id/restore`, `/:id/download`
+ (302 for public, HMAC-signed short-lived URL for private —
+ `MEDIA_SIGNING_SECRET`). App key, or a Network service token with
+ `media.object.upload` / `media.object.read` for namespace `:app`;
+ `X-OV-Subject` sets the owner.
+- **Retention holds** — `media_holds` (`moderation`, `dmca`, `creator_pin`,
+ `admin`, `evidence`; `/:id/holds`, app key only): a held object cannot be
+ deleted by any path (409 / DB trigger) or moved between tiers.
+- **Reconciliation** — `node scripts/reconcile-objects.js [--verify] [--hash]`:
+ read-only by default; `--verify` HEADs B2/R2 and records location states.
+ Reports missing canonical copies, lost local files, size/hash mismatches,
+ orphan locations, deleted-but-still-served objects and unprojected rows.
+- **Public object-size invariant** — `MEDIA_PUBLIC_OBJECT_MAX_MB` (500) /
+ `_TARGET_MB` (256) / `_WARN_MB` (384): v2 refuses oversized public playback
+ uploads; `node scripts/object-invariant.js` lists offenders and records them in
+ `media_invariant_violations`. No automatic re-encoding.
+
## Storage tiering
## Storage tiering
`vod/vod-storage.js` (ported nearly as-is): local → B2 canonical cold tier →
`vod/vod-storage.js` (ported nearly as-is): local → B2 canonical cold tier →
new file mode 100644
new file mode 100644
@@ -0,0 +1,258 @@
@@ -0,0 +1,258 @@
+# Object model
+
+Roadmap Wave 4 turns Media from predecessor-shaped VOD/clip/file tables into one canonical object
+platform. Every stored blob is a **media object** with an id `med_<ULID>`. The inherited
+`vods`, `clips`, `files` and screenshot `pastes` rows stay as they are, and each one is now a typed
+projection over an object through its `object_id` column. Every existing URL and v1 response is
+unchanged. New code talks to objects through the [v2 API](#object-api-v2).
+
+Code: `server/objects/` (`model.js`, `routes.js`, `backfill.js`, `reconcile.js`, `invariant.js`,
+`signing.js`). Schema: the bottom of `server/db/schema.sql`. Tests: `test/objects-*.test.js`.
+
+## Tables
+
+| table | what it holds |
+|---|---|
+| `media_objects` | `id` (med_…), `app_id` (tenant), `namespace` (capability namespace; = app today), `kind` (`vod` `clip` `file` `thumbnail` `screenshot` `avatar` `asset`), `owner_subject` (usr_… when known), `owner_app` + `owner_user_id` (legacy owner in the app's own user-id space), `visibility` (`public` `unlisted` `private`), `lifecycle_status` (`uploading` `ready` `failed` `archived` `deleted`), `mime_type`, `size_bytes`, `content_hash` (sha256), `canonical_provider` + `canonical_key`, `legacy_ref`, `metadata` (JSON), `created_at` / `updated_at` / `deleted_at` |
+| `media_locations` | one row per provider copy (`UNIQUE(object_id, provider)`): `provider` (`local` `b2` `r2`), `bucket`, `key` (absolute path for local, object key for B2/R2), `storage_class` (`hot` local, `cold` B2, `cache` R2), `state`, `checksum`, `size_bytes`, `verified_at` |
+| `media_relationships` | `from_object_id`, `relation`, `to_object_id`, `metadata`. In use: `clip_of` (clip to vod, with start/end), `thumbnail_of` (thumbnail to vod/clip). Reserved: `derived_from`, `screenshot_of` |
+| `media_variants` | `object_id`, `variant_name`, `derived_object_id`, `recipe`. In use: `thumbnail` |
+| `media_jobs` | generic derivative/maintenance jobs (`job_type`, `status`, `attempts`, `checkpoint`, `error`). **Schema only in this pass:** nothing enqueues or consumes jobs yet |
+| `media_holds` | retention holds, see [Holds](#retention-holds) |
+| `media_invariant_violations` | public playback objects over the size policy, see [Invariant](#public-object-size-invariant) |
+
+Location `state`:
+
+- `present`: verified. The local file exists, or a HEAD answered with the right size.
+- `missing`: verified absent.
+- `pending`: believed there but not verified yet. Every remote copy starts here until reconciliation runs with `--verify`.
+- `corrupt`: the size or hash does not match.
+
+A copy that reconciliation has verified keeps that state when the row is projected again. The
+state resets only when the key changes.
+
+### Ids and legacy refs
+
+Ids come from `openvibe-contracts` `ids.newId('media')`. A backfilled object takes its ULID time
+from the original row's `created_at`, so ids sort by creation time. Projected objects also carry
+the transitional `media.media-ref@1` legacy form in `legacy_ref`, which is unique:
+
+| row | `kind` | `legacy_ref` |
+|---|---|---|
+| `vods.id` | vod | `legacy:<app>:vod:<id>` |
+| `clips.id` | clip | `legacy:<app>:clip:<id>` |
+| `files.key` | file | `legacy:<app>:file:<key>` |
+| screenshot paste | screenshot | `legacy:<app>:paste:<slug>` |
+| avatar (screenshot paste with `metadata.kind = 'avatar'`) | avatar | `legacy:<app>:avatar:<slug>` |
+| `vods`/`clips.thumbnail_url` (`/t/<name>`) | thumbnail | `legacy:<app>:thumbnail:<name>` |
+
+Every v2 route that takes `:id` accepts either form. Native v2 objects have `legacy_ref = NULL`.
+
+### How a row maps to its object
+
+`model.sync(kind, id)` derives the object and its locations from the row as it is at that moment.
+The backfill and every write hook use the same function.
+
+- **vod / clip.**
+ - Local copy: the file under `VOD_PATH` (vods, resolved by basename as playback does) or the clip's own path.
+ - B2 copy (canonical, key `vods/<basename>` or `storage_key`): added when `storage_provider` is `b2`/`r2` or a `storage_key` is set.
+ - R2 copy (cache): added when the provider is `r2`.
+ - Canonical copy: B2 when one exists, otherwise local.
+ - Lifecycle: recording → `uploading`; `corrupt`/`zero_byte`/`missing_file` → `failed`; `ready` otherwise. Clips: `processing` → `uploading`, `failed` → `failed`.
+ - Visibility: `visibility`, falling back to `is_public`, the same way playback decides.
+ - Clips-only (ephemeral) recordings are never projected.
+- **file.** The local file under `FILES_PATH/<app>/<key>`. `visibility = public`, because `/f/:key` serves every file without auth. The row's sha256 becomes both `content_hash` and the location `checksum`.
+- **screenshot / avatar.** `screenshot_path`. Text pastes have no bytes and no object; paste text belongs to Community.
+- **thumbnail.** One object per vod/clip, reached through the parent's `thumbnail` variant and **updated in place** when the picture is regenerated (live recordings refresh it every couple of minutes). Visibility is `public` when the parent is public and `unlisted` otherwise, because `/t/<name>` serves anyone who has the name. An external `thumbnail_url` is skipped.
+
+## Backfill
+
+```
+node scripts/backfill-objects.js [--dry-run] [--only-missing] [--json] [--out report.json] [--db ./data/media.db]
+```
+
+The backfill creates one object per vod, clip, file, screenshot/avatar paste and thumbnail, with
+relationships and thumbnail variants.
+
+- **Idempotent.** Objects are keyed by `legacy_ref`, so a second run updates them in place and creates nothing.
+- **Never moves or deletes bytes, and never calls B2/R2.** A local copy is `present` when the file exists and `missing` when it does not. Remote copies stay `pending` until `reconcile-objects.js --verify` checks them.
+- **One transaction,** with one savepoint per row: a failing row is reported and leaves nothing half-written. `--dry-run` rolls everything back and only reports.
+- **Report:** counts per kind (seen, created, updated, skipped), location states, skipped rows with a reason, and errors. A real run stores the report in `media_settings` under `objects.backfill.last_report`.
+- **At boot:** the service runs the `--only-missing` form itself 15 s after start, for rows with no `object_id`. The first boot after the upgrade therefore projects everything.
+
+Skip reasons today: `clips-only recording (ephemeral, never published)`, `screenshot paste without a file path`, `external thumbnail url`.
+
+The run holds SQLite's write lock for its duration, typically seconds. Prefer the boot backfill, or
+run the script while traffic is quiet.
+
+### Keeping the model current
+
+The old write paths re-project their row after they change it:
+
+| path | hook |
+|---|---|
+| vod create | `POST /vods` |
+| recording start | `recorder` sets `file_path` |
+| every finalize outcome | `finalize.finalizeVod` |
+| visibility | `db.setVodVisibility` / `setClipVisibility` |
+| health | `db.updateVodHealth` |
+| title | `PUT /vods/:id`, `PUT /clips/:id` |
+| clip create | `db.createClip` (API cuts, uploads and auto-clips) |
+| clip cut / re-cut result | clips routes and `clip-jobs` |
+| file upload | `db.createFile` |
+| thumbnail generate / upload | `thumbnail-service`, `POST /thumbnails` |
+| screenshot upload / censor | pastes routes |
+| avatar ingest | `avatars/ingest` |
+| tier moves | `moveToCold`, `moveToHot`, `promoteToR2`, `demoteFromR2`, sweep, `migrateLegacy`: the copy the move verified is marked `present` |
+
+**Deletes need no hook.** A SQLite trigger on `vods`, `clips`, `files` and `pastes` marks the
+object `deleted` whenever its row is deleted, which covers all of the inherited delete paths. Hooks
+never throw: the legacy write has already happened, so a failed sync only logs a warning. The next
+sync or backfill repairs it.
+
+## Object API v2
+
+All routes live under `/api/v2/:app/objects`.
+
+**Auth.** One of:
+
+- the app's API key;
+- a Network service token that grants `media.object.upload` (writes) or `media.object.read` (reads) for the namespace `:app`. `tenantAuth({ capability })` enforces this, as it does on v1.
+
+**Owner.**
+
+- `X-OV-Subject: usr_…` (or `user:usr_…`) sets `owner_subject`.
+- With an app key, `X-OV-User-Id` acts as one of the app's users. That caller sees public and unlisted objects plus its own, and can only write its own.
+
+**Errors** are RFC 9457 problem bodies (`openvibe-contracts` `http.sendProblem`).
+
+| method | path | notes |
+|---|---|---|
+| POST | `/` | init. `{ kind, visibility (default private), mime_type, size_bytes, filename, content_hash (expected sha256), metadata (≤ 16 KB), user_id }` returns 201 `{ id, object, upload: { method: 'PUT', url, token, expires_at, max_bytes, complete_url } }`. Checks at init: `size_bytes` against `MEDIA_OBJECT_MAX_MB` (413), the quota including reservations (413 `media.quota.exceeded`), and the public-size invariant (422) |
+| PUT | `/:id/content` | the bytes, single part. Auth is the upload token (`?token=` from init, or `X-Upload-Token`) or the usual credential. Streams to disk computing sha256. Refuses more than the declared size or the limit (413), a size mismatch (400) and quota overrun (413). Stored at `OBJECTS_PATH/<app>/<id>` with a verified local location. Mounted ahead of the JSON body parser, so any Content-Type is taken as raw bytes; the Content-Type becomes `mime_type` when init set none. Can be repeated while the object is `uploading` |
+| POST | `/:id/complete` | checks the expected hash (422 `media.object.hash_mismatch`), the invariant and the quota. Sets `ready` and sends the `media.object.uploaded` webhook. Also accepts the upload token |
+| GET | `/:id` | metadata: providers and states of each copy, never paths or keys |
+| GET | `/` | cursor list, newest first. `?limit (≤200)&cursor&kind&visibility&status&owner (usr_…)&user_id&include_deleted` returns `{ objects, next_cursor }` |
+| DELETE | `/:id` | soft delete: `lifecycle_status = deleted`, and the bytes are kept for `MEDIA_DELETE_RETENTION_DAYS` (default 30). 409 `media.object.held` under a hold. 409 `media.object.legacy_managed` for projected objects, which are deleted through their v1 route |
+| POST | `/:id/restore` | undo a soft delete within the retention period (410 once purged) |
+| GET | `/:id/download` | public or unlisted: 302 to the public location (the inherited URL for projected objects such as `/v/:id`, `/c/:id`, `/f/:key`, `/t/:name`, `/p/:slug/screenshot`; otherwise `/o/:id`). Private: `{ url, expires_at }` signed for `?ttl` seconds (30 to 3600, default `MEDIA_SIGNED_URL_TTL_S` = 300), or a 302 with `?redirect=1`. `?format=json` always answers JSON. 410 when deleted |
+| GET | `/:id/holds` | active holds (`?all=1` includes released ones) |
+| POST | `/:id/holds` | **app key only.** `{ kind, reason, created_by }` |
+| DELETE | `/:id/holds/:holdId` | **app key only.** Releases the hold (`released_at`, `released_by`) |
+
+**Public bytes: `GET /o/:id`.**
+
+- Public and unlisted `ready` objects are served openly.
+- Private objects need a valid `?exp=&sig=` from `/download`. Without one the answer is 404, so a private object cannot be told apart from a missing one.
+- A deleted object answers 410.
+- A local copy streams with Range support. Otherwise the route redirects 302 to a presigned R2 URL, then a presigned B2 URL.
+- Headers: `X-Content-Type-Options: nosniff` and `X-Robots-Tag: noindex`. Content is `inline` only for images (not SVG), video, audio, PDF and plain text, and `attachment` for everything else.
+
+**Signing.** HMAC-SHA256 over the purpose (`get` or `put`), the object id and the expiry, keyed with
+`MEDIA_SIGNING_SECRET`. A download signature cannot be used as an upload token, or the other way
+round. Without the secret, a random per-process secret is used and a warning is logged, so links
+stop working at the next restart. **Set `MEDIA_SIGNING_SECRET` in production.**
+
+**Quota.** `apps.quota_bytes` is compared against v1 file bytes plus native objects that are
+`uploading` (their declared size counts as a reservation) or `ready`. Soft-deleted objects do not
+count. The v1 files route still counts only v1 files, so its response is unchanged.
+
+**Purge.** Every hour, the service removes the bytes of native objects whose retention period has
+passed and that have no active hold. The object row stays `deleted` with `metadata.purged_at`.
+Projected objects are never purged by this job.
+
+## Retention holds
+
+`media_holds(object_id, kind, reason, created_by, created_at, released_at, released_by)`. The
+`kind` is one of `moderation`, `dmca`, `creator_pin`, `admin`, `evidence`. While any hold is
+unreleased:
+
+- **No delete path works.**
+ - v2 `DELETE` answers 409.
+ - The v1 deletes for vods, clips and files, and the admin bulk delete, answer 409 or report `held`.
+ - Quarantine cleanup skips the object.
+ - `deleteVodObjects` keeps the bytes.
+ - Paste screenshot removal keeps the file, and censor answers 409.
+ - A `BEFORE DELETE` trigger on each projected table, plus a trigger on `media_objects.lifecycle_status`, refuses the change, so paths nobody has hooked are still covered.
+- **No tier move.** `moveToCold`, `promoteToR2` and `demoteFromR2` return `{ ok: false, held: true }`, and the sweep counts these as `skippedHeld` instead of errors. A hold freezes an object's placement. `moveToHot` (restoring a copy to local disk) is still allowed.
+
+## Reconciliation
+
+```
+node scripts/reconcile-objects.js [--verify] [--hash] [--app live] [--json] [--out report.json] [--db ./data/media.db]
+```
+
+**Modes.**
+
+- **Default: read-only.** Checks local files and database consistency, writes nothing and calls no provider.
+- `--verify`: also HEADs every B2/R2 location with the storage engine's S3 client, and records what it finds on `media_locations` (`state`, `size_bytes`, `verified_at`). An unconfigured provider counts as *unverifiable*, never as *missing*.
+- `--hash`: computes the sha256 of local files ≤ 512 MB for objects that carry a `content_hash`.
+
+The exit code is 1 when issues were found.
+
+**Issues reported** (each with a count and up to 200 examples):
+
+| issue | meaning |
+|---|---|
+| `no_canonical_location` | a ready object with no location row for its canonical provider |
+| `canonical_missing_replica_present` | the canonical copy is missing or corrupt while another copy is present (for example B2 lost it but R2 has it) |
+| `no_present_copy` | a ready object whose every copy is missing or corrupt |
+| `missing_local_file` | a ready object's local copy is not on disk |
+| `remote_missing` | a ready object's B2/R2 copy answered 404 (`--verify`) |
+| `size_mismatch` / `hash_mismatch` | the bytes differ from the object's size or sha256 |
+| `orphan_location` | a location row whose object does not exist |
+| `deleted_publicly_reachable` | a deleted object whose inherited row still serves it, or a thumbnail of a deleted object that is still served by name |
+| `missing_projection` | an inherited row with no object yet (run the backfill) |
+
+**Not covered yet:** orphan *provider* objects (bucket keys with no row, which needs bucket
+listing) and incomplete multipart uploads. `server/objects/reconcile.js` takes a `head` function,
+so tests run it against a fake provider.
+
+## Public object-size invariant
+
+This is roadmap W4 deliverable 5, stated as **policy** rather than a copied constant:
+
+| setting | env | default |
+|---|---|---|
+| max: no public playback object above this | `MEDIA_PUBLIC_OBJECT_MAX_MB` | 500 |
+| target | `MEDIA_PUBLIC_OBJECT_TARGET_MB` | 256 |
+| warn | `MEDIA_PUBLIC_OBJECT_WARN_MB` | 384 |
+
+A *public playback object* is a `ready` vod or clip whose visibility is `public` or `unlisted`.
+
+- **Validator.** `server/objects/invariant.js` classifies a size as `ok`, `above_target`, `warn` or `violation`. The functions are pure.
+- **Upload time.** v2 refuses to init or complete a public playback object above the max (422 `media.invariant.public_object_too_large`). Private objects are not playback objects.
+- **Violations table.** Every vod/clip sync records warn and violation levels in `media_invariant_violations`, one row per object. The row is resolved when the object shrinks, becomes private or is deleted.
+- **Report.**
+
+ ```
+ node scripts/object-invariant.js [--dry-run] [--app live] [--json] [--db ./data/media.db]
+ ```
+
+ Lists every public playback object above target, largest first, records the rows, and resolves rows that no longer apply. `--dry-run` writes nothing.
+
+**Nothing is re-encoded in this pass.** Most whole-stream VODs are far above 500 MB. They will show
+up as violations until the segment-native model lands (roadmap W4 deliverable 7, capture in W7),
+which splits playback into objects that fit.
+
+## Configuration
+
+| env | default | meaning |
+|---|---|---|
+| `OBJECTS_PATH` | `./data/objects` | native object bytes (`<app>/<id>`, `.tmp/` while uploading) |
+| `MEDIA_OBJECT_MAX_MB` | 256 | single-part upload limit |
+| `MEDIA_DELETE_RETENTION_DAYS` | 30 | soft-deleted native bytes are kept this long |
+| `MEDIA_SIGNING_SECRET` | *(random per process)* | HMAC key for signed URLs and upload tokens |
+| `MEDIA_SIGNED_URL_TTL_S` | 300 | default signed download lifetime |
+| `MEDIA_UPLOAD_TOKEN_TTL_S` | 3600 | upload token lifetime |
+| `MEDIA_PUBLIC_OBJECT_{MAX,TARGET,WARN}_MB` | 500 / 256 / 384 | invariant thresholds |
+
+## Not in this pass
+
+- Multipart uploads and presigned direct-to-bucket uploads. Native bytes are stored locally.
+- Tiering of native objects to B2/R2.
+- Copy/move aliases, lifecycle rules and an S3-compatible façade.
+- A `media_jobs` worker.
+- Orphan-bucket-object and incomplete-multipart detection.
+- The segment-native timeline.
+- App assets (emotes and sounds, the `assets` table) are not projected yet. The `asset` kind exists for v2 uploads.
new file mode 100755
new file mode 100755
@@ -0,0 +1,34 @@
@@ -0,0 +1,34 @@
+#!/usr/bin/env node
+'use strict';
+/**
+ * Project every existing vod, clip, file, screenshot/avatar paste and thumbnail onto the canonical
+ * object model (media_objects + media_locations; docs/object-model.md). Idempotent — re-runs update
+ * in place. Never moves or deletes bytes and never calls B2/R2 (remote copies stay 'pending' until
+ * scripts/reconcile-objects.js --verify).
+ *
+ * node scripts/backfill-objects.js [--dry-run] [--only-missing] [--json] [--out report.json] [--db ./data/media.db]
+ *
+ * The service runs the --only-missing form itself shortly after boot. The run is one transaction
+ * (a dry run rolls it back), so it holds SQLite's write lock for its duration — seconds, typically.
+ */
+const fs = require('fs');
+const path = require('path');
+
+const args = process.argv.slice(2);
+const has = (name) => args.includes(`--${name}`);
+const arg = (name, dflt) => { const i = args.indexOf(`--${name}`); return i >= 0 ? args[i + 1] : dflt; };
+if (arg('db')) process.env.DB_PATH = path.resolve(arg('db'));
+
+const { backfill, summarize } = require('../server/objects/backfill');
+const report = backfill({ dryRun: has('dry-run'), onlyMissing: has('only-missing') });
+if (arg('out')) fs.writeFileSync(arg('out'), JSON.stringify(report, null, 2));
+if (has('json')) console.log(JSON.stringify(report, null, 2));
+else {
+ console.log(summarize(report));
+ const reasons = {};
+ for (const s of report.skipped) reasons[s.reason] = (reasons[s.reason] || 0) + 1;
+ for (const [r, n] of Object.entries(reasons)) console.log(` skipped ${n}: ${r}`);
+ for (const e of report.errors.slice(0, 20)) console.log(` error ${e.table} ${e.id}: ${e.error}`);
+}
+require('../server/db/database').close();
+process.exit(report.errors.length ? 1 : 0);
new file mode 100755
new file mode 100755
@@ -0,0 +1,32 @@
@@ -0,0 +1,32 @@
+#!/usr/bin/env node
+'use strict';
+/**
+ * Public object-size invariant report (roadmap W4 deliverable 5; docs/object-model.md#public-object-size-invariant).
+ * Lists public/unlisted ready VOD and clip objects above MEDIA_PUBLIC_OBJECT_TARGET_MB (256), flags
+ * those above MEDIA_PUBLIC_OBJECT_WARN_MB (384) and MEDIA_PUBLIC_OBJECT_MAX_MB (500), and records
+ * warn/violation rows in media_invariant_violations (resolving rows that no longer apply).
+ * Nothing is re-encoded.
+ *
+ * node scripts/object-invariant.js [--dry-run] [--app live] [--json] [--db ./data/media.db]
+ */
+const path = require('path');
+
+const args = process.argv.slice(2);
+const has = (name) => args.includes(`--${name}`);
+const arg = (name, dflt) => { const i = args.indexOf(`--${name}`); return i >= 0 ? args[i + 1] : dflt; };
+if (arg('db')) process.env.DB_PATH = path.resolve(arg('db'));
+
+const { scan, MB } = require('../server/objects/invariant');
+const report = scan({ record: !has('dry-run'), appId: arg('app', null) });
+if (has('json')) console.log(JSON.stringify(report, null, 2));
+else {
+ const t = report.thresholds;
+ console.log(`public playback objects — target ${t.targetBytes / MB} MB, warn ${t.warnBytes / MB} MB, max ${t.maxBytes / MB} MB${has('dry-run') ? ' (dry run)' : ''}`);
+ console.log(` ok ${report.counts.ok}, above target ${report.counts.above_target}, warn ${report.counts.warn}, violation ${report.counts.violation}`);
+ for (const o of report.objects.slice(0, 50)) {
+ console.log(` ${o.level.padEnd(12)} ${(o.size_bytes / MB).toFixed(0).padStart(6)} MB ${o.object_id} ${o.legacy_ref || ''}`);
+ }
+ if (report.objects.length > 50) console.log(` … ${report.objects.length - 50} more (use --json)`);
+}
+require('../server/db/database').close();
+process.exit(0);
new file mode 100755
new file mode 100755
@@ -0,0 +1,31 @@
@@ -0,0 +1,31 @@
+#!/usr/bin/env node
+'use strict';
+/**
+ * Reconcile the object model against the bytes (docs/object-model.md#reconciliation).
+ *
+ * node scripts/reconcile-objects.js [--verify] [--hash] [--app live] [--json] [--out report.json] [--db ./data/media.db]
+ *
+ * Default: read-only — checks local files and database consistency, writes nothing.
+ * --verify also HEADs every B2/R2 location with the storage engine's S3 client (MEDIA_B2_* / MEDIA_R2_*)
+ * and records the result on media_locations (state, size, verified_at).
+ * --hash sha256 local files of objects that carry a content hash (≤ 512 MB each).
+ * Exit code 0 when clean, 1 when issues were found.
+ */
+const fs = require('fs');
+const path = require('path');
+
+const args = process.argv.slice(2);
+const has = (name) => args.includes(`--${name}`);
+const arg = (name, dflt) => { const i = args.indexOf(`--${name}`); return i >= 0 ? args[i + 1] : dflt; };
+if (arg('db')) process.env.DB_PATH = path.resolve(arg('db'));
+
+const { reconcile, summarize } = require('../server/objects/reconcile');
+
+(async () => {
+ if (has('verify')) await require('../server/vod/vod-storage').checkProviders().catch(() => {});
+ const report = await reconcile({ verify: has('verify'), hash: has('hash'), appId: arg('app', null) });
+ if (arg('out')) fs.writeFileSync(arg('out'), JSON.stringify(report, null, 2));
+ console.log(has('json') ? JSON.stringify(report, null, 2) : summarize(report));
+ require('../server/db/database').close();
+ process.exit(report.total_issues ? 1 : 0);
+})().catch((err) => { console.error(err); process.exit(2); });
@@ -214,6 +214,7 @@ router.delete('/vods/bulk', async (req, res) => {
@@ -214,6 +214,7 @@ router.delete('/vods/bulk', async (req, res) => {
try {
try {
const vod = Number.isFinite(id) ? db.getVodById(id, req.appId) : null;
const vod = Number.isFinite(id) ? db.getVodById(id, req.appId) : null;
if (!vod) { result.error = 'VOD not found'; results.push(result); continue; }
if (!vod) { result.error = 'VOD not found'; results.push(result); continue; }
+ if (require('../objects/model').isHeldRow(vod)) { result.error = 'VOD is under a retention hold'; results.push(result); continue; }
if (recorder.isRecording(vod.id)) recorder.stopRecording(vod.id);
if (recorder.isRecording(vod.id)) recorder.stopRecording(vod.id);
try { require('../vod/routes').activeChunkUploads.delete(vod.id); } catch { /* */ }
try { require('../vod/routes').activeChunkUploads.delete(vod.id); } catch { /* */ }
@@ -118,10 +118,11 @@ function createIngestHandler({ db, config, screenshotsDir, generateSlug, log = c
@@ -118,10 +118,11 @@ function createIngestHandler({ db, config, screenshotsDir, generateSlug, log = c
const file = path.join(screenshotsDir, `avatar-n${parseInt(user_id, 10) || 0}-${Date.now()}-${crypto.randomBytes(4).toString('hex')}.webp`);
const file = path.join(screenshotsDir, `avatar-n${parseInt(user_id, 10) || 0}-${Date.now()}-${crypto.randomBytes(4).toString('hex')}.webp`);
fs.writeFileSync(file, pic.buffer);
fs.writeFileSync(file, pic.buffer);
const slug = generateSlug();
const slug = generateSlug();
- db.run(`INSERT INTO pastes (app_id, slug, user_id, type, title, content, language, visibility, screenshot_path, metadata, ip_address)
+ const info = db.run(`INSERT INTO pastes (app_id, slug, user_id, type, title, content, language, visibility, screenshot_path, metadata, ip_address)
VALUES ('network', ?, ?, 'screenshot', ?, '', 'text', 'unlisted', ?, ?, NULL)`,
VALUES ('network', ?, ?, 'screenshot', ?, '', 'text', 'unlisted', ?, ?, NULL)`,
[slug, parseInt(user_id, 10) || null, `Avatar${username ? ' of ' + String(username).slice(0, 40) : ''}`, file,
[slug, parseInt(user_id, 10) || null, `Avatar${username ? ' of ' + String(username).slice(0, 40) : ''}`, file,
JSON.stringify({ kind: 'avatar', source_host: (() => { try { return new URL(url).hostname; } catch { return null; } })(), size_bytes: pic.buffer.length, mime_type: 'image/webp' })]);
JSON.stringify({ kind: 'avatar', source_host: (() => { try { return new URL(url).hostname; } catch { return null; } })(), size_bytes: pic.buffer.length, mime_type: 'image/webp' })]);
+ if (info && typeof db.syncObject === 'function') db.syncObject('paste', info.lastInsertRowid); // the avatar's media_object
res.json({ ok: true, slug, url: `${config.publicUrl}/p/${slug}/screenshot`, width: pic.width, height: pic.height, bytes: pic.buffer.length });
res.json({ ok: true, slug, url: `${config.publicUrl}/p/${slug}/screenshot`, width: pic.width, height: pic.height, bytes: pic.buffer.length });
} catch (err) {
} catch (err) {
log.warn('[Avatar ingest]', err.message);
log.warn('[Avatar ingest]', err.message);
@@ -43,6 +43,20 @@ const config = {
@@ -43,6 +43,20 @@ const config = {
maxSizeMb: intEnv('MAX_FILE_SIZE_MB', 256),
maxSizeMb: intEnv('MAX_FILE_SIZE_MB', 256),
},
},
+ // Canonical object model (Wave 4): native v2 uploads, signed delivery, retention,
+ // and the public playback object-size invariant (policy, not a copied constant).
+ objects: {
+ path: process.env.OBJECTS_PATH || './data/objects',
+ maxUploadMb: intEnv('MEDIA_OBJECT_MAX_MB', 256), // single-part upload limit
+ retentionDays: intEnv('MEDIA_DELETE_RETENTION_DAYS', 30), // soft-deleted bytes kept this long
+ signingSecret: process.env.MEDIA_SIGNING_SECRET || '',
+ signedUrlTtlS: intEnv('MEDIA_SIGNED_URL_TTL_S', 300),
+ uploadTokenTtlS: intEnv('MEDIA_UPLOAD_TOKEN_TTL_S', 3600),
+ publicMaxMb: intEnv('MEDIA_PUBLIC_OBJECT_MAX_MB', 500),
+ publicTargetMb: intEnv('MEDIA_PUBLIC_OBJECT_TARGET_MB', 256),
+ publicWarnMb: intEnv('MEDIA_PUBLIC_OBJECT_WARN_MB', 384),
+ },
+
rtp: {
rtp: {
portMin: intEnv('RTP_PORT_MIN', 12000),
portMin: intEnv('RTP_PORT_MIN', 12000),
portMax: intEnv('RTP_PORT_MAX', 12199),
portMax: intEnv('RTP_PORT_MAX', 12199),
@@ -71,6 +85,7 @@ config.pastes.path = path.resolve(config.pastes.path);
@@ -71,6 +85,7 @@ config.pastes.path = path.resolve(config.pastes.path);
config.assets.path = path.resolve(config.assets.path);
config.assets.path = path.resolve(config.assets.path);
config.thumbnails.path = path.resolve(config.thumbnails.path);
config.thumbnails.path = path.resolve(config.thumbnails.path);
config.files.path = path.resolve(config.files.path);
config.files.path = path.resolve(config.files.path);
+config.objects.path = path.resolve(config.objects.path);
config.db.path = path.resolve(config.db.path);
config.db.path = path.resolve(config.db.path);
module.exports = config;
module.exports = config;
@@ -28,6 +28,7 @@ function getDb() {
@@ -28,6 +28,7 @@ function getDb() {
const schema = fs.readFileSync(path.join(__dirname, 'schema.sql'), 'utf8');
const schema = fs.readFileSync(path.join(__dirname, 'schema.sql'), 'utf8');
database.exec(schema);
database.exec(schema);
migrateColumns();
migrateColumns();
+ ensureObjectTriggers();
normalizeThumbnailUrls();
normalizeThumbnailUrls();
seedSettings();
seedSettings();
recoverInterruptedClips();
recoverInterruptedClips();
@@ -71,8 +72,10 @@ function normalizeThumbnailUrls() {
@@ -71,8 +72,10 @@ function normalizeThumbnailUrls() {
// (CREATE TABLE IF NOT EXISTS won't add columns to an existing table).
// (CREATE TABLE IF NOT EXISTS won't add columns to an existing table).
function migrateColumns() {
function migrateColumns() {
const wanted = {
const wanted = {
- vods: [['managed_stream_id', 'INTEGER']],
- clips: [['channel_user_id', 'INTEGER'], ['cut_error', 'TEXT'], ['cut_attempts', 'INTEGER DEFAULT 0'], ['cut_next_at', 'DATETIME']],
+ vods: [['managed_stream_id', 'INTEGER'], ['object_id', 'TEXT']],
+ clips: [['channel_user_id', 'INTEGER'], ['cut_error', 'TEXT'], ['cut_attempts', 'INTEGER DEFAULT 0'], ['cut_next_at', 'DATETIME'], ['object_id', 'TEXT']],
+ files: [['object_id', 'TEXT']],
+ pastes: [['object_id', 'TEXT']],
};
};
for (const [table, cols] of Object.entries(wanted)) {
for (const [table, cols] of Object.entries(wanted)) {
const existing = database.prepare(`PRAGMA table_info(${table})`).all().map(c => c.name);
const existing = database.prepare(`PRAGMA table_info(${table})`).all().map(c => c.name);
@@ -84,6 +87,43 @@ function migrateColumns() {
@@ -84,6 +87,43 @@ function migrateColumns() {
}
}
}
}
+/**
+ * Object-model guards that must hold on EVERY delete path, including the many
+ * inherited ones that DELETE a vods/clips/files/pastes row directly:
+ * - a row whose object is under an unreleased retention hold cannot be deleted;
+ * - deleting a projected row marks its media_object deleted (bytes accounting
+ * and reconciliation keep working without touching each call site).
+ * Created here, after migrateColumns, because they reference object_id.
+ */
+function ensureObjectTriggers() {
+ const held = (ref) => `EXISTS (SELECT 1 FROM media_holds WHERE object_id = ${ref} AND released_at IS NULL)`;
+ for (const table of ['vods', 'clips', 'files', 'pastes']) {
+ database.exec(`
+ CREATE TRIGGER IF NOT EXISTS trg_${table}_hold_guard BEFORE DELETE ON ${table}
+ WHEN OLD.object_id IS NOT NULL AND ${held('OLD.object_id')}
+ BEGIN SELECT RAISE(ABORT, 'media object is under a retention hold'); END;
+ CREATE TRIGGER IF NOT EXISTS trg_${table}_object_deleted AFTER DELETE ON ${table}
+ WHEN OLD.object_id IS NOT NULL
+ BEGIN
+ UPDATE media_objects SET lifecycle_status = 'deleted', deleted_at = COALESCE(deleted_at, CURRENT_TIMESTAMP),
+ updated_at = CURRENT_TIMESTAMP
+ WHERE id = OLD.object_id AND lifecycle_status != 'deleted';
+ END;`);
+ }
+ database.exec(`
+ CREATE TRIGGER IF NOT EXISTS trg_media_objects_hold_guard BEFORE UPDATE OF lifecycle_status ON media_objects
+ WHEN NEW.lifecycle_status = 'deleted' AND OLD.lifecycle_status != 'deleted' AND ${held('OLD.id')}
+ BEGIN SELECT RAISE(ABORT, 'media object is under a retention hold'); END;
+ CREATE TRIGGER IF NOT EXISTS trg_media_objects_hold_delete BEFORE DELETE ON media_objects
+ WHEN ${held('OLD.id')}
+ BEGIN SELECT RAISE(ABORT, 'media object is under a retention hold'); END;`);
+}
+
+/** Keep a projected row's media_object current. Never throws — the legacy write already happened. */
+function _syncObject(kind, id) {
+ return require('../objects/model').safeSync(kind, id);
+}
+
function seedSettings() {
function seedSettings() {
const defaults = [
const defaults = [
// Clip system
// Clip system
@@ -317,7 +357,9 @@ function _normVisibility(v) { return VALID_VISIBILITY.has(v) ? v : 'public'; }
@@ -317,7 +357,9 @@ function _normVisibility(v) { return VALID_VISIBILITY.has(v) ? v : 'public'; }
// Set VOD/clip visibility; is_public mirrors (1 iff public) so listing filters hold.
// Set VOD/clip visibility; is_public mirrors (1 iff public) so listing filters hold.
function setVodVisibility(vodId, visibility) {
function setVodVisibility(vodId, visibility) {
const vis = _normVisibility(visibility);
const vis = _normVisibility(visibility);
- return run('UPDATE vods SET visibility = ?, is_public = ? WHERE id = ?', [vis, vis === 'public' ? 1 : 0, vodId]);
+ const r = run('UPDATE vods SET visibility = ?, is_public = ? WHERE id = ?', [vis, vis === 'public' ? 1 : 0, vodId]);
+ _syncObject('vod', vodId);
+ return r;
}
}
function updateVodHealth(vodId, { status, score, issues = [], probeDuration, probeFormat, quarantine = false, keepPublic = false }) {
function updateVodHealth(vodId, { status, score, issues = [], probeDuration, probeFormat, quarantine = false, keepPublic = false }) {
@@ -335,7 +377,9 @@ function updateVodHealth(vodId, { status, score, issues = [], probeDuration, pro
@@ -335,7 +377,9 @@ function updateVodHealth(vodId, { status, score, issues = [], probeDuration, pro
updates.push("last_health_scan_at = datetime('now')");
updates.push("last_health_scan_at = datetime('now')");
params.push(vodId);
params.push(vodId);
if (!updates.length) return null;
if (!updates.length) return null;
- return run(`UPDATE vods SET ${updates.join(', ')} WHERE id = ?`, params);
+ const r = run(`UPDATE vods SET ${updates.join(', ')} WHERE id = ?`, params);
+ _syncObject('vod', vodId);
+ return r;
}
}
function repairVodDuration(vodId, duration, fileSize) {
function repairVodDuration(vodId, duration, fileSize) {
@@ -380,13 +424,15 @@ function vodStatus(vod) {
@@ -380,13 +424,15 @@ function vodStatus(vod) {
function createClip({ app_id, vod_id, stream_id, user_id, channel_user_id, title, description, file_path, thumbnail_url, start_time, end_time, duration_seconds, is_public, auto_generated, status }) {
function createClip({ app_id, vod_id, stream_id, user_id, channel_user_id, title, description, file_path, thumbnail_url, start_time, end_time, duration_seconds, is_public, auto_generated, status }) {
const pub = (is_public === 0 || is_public === false) ? 0 : 1;
const pub = (is_public === 0 || is_public === false) ? 0 : 1;
- return run(
+ const r = run(
`INSERT INTO clips (app_id, vod_id, stream_id, user_id, channel_user_id, title, description, file_path, thumbnail_url, start_time, end_time, duration_seconds, is_public, visibility, auto_generated, status)
`INSERT INTO clips (app_id, vod_id, stream_id, user_id, channel_user_id, title, description, file_path, thumbnail_url, start_time, end_time, duration_seconds, is_public, visibility, auto_generated, status)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
[app_id, vod_id || null, stream_id || null, user_id || null, channel_user_id || null, title || 'Untitled Clip', description || '',
[app_id, vod_id || null, stream_id || null, user_id || null, channel_user_id || null, title || 'Untitled Clip', description || '',
file_path || '', thumbnail_url || null, start_time || 0, end_time || 0, duration_seconds || 0,
file_path || '', thumbnail_url || null, start_time || 0, end_time || 0, duration_seconds || 0,
pub, pub ? 'public' : 'unlisted', auto_generated ? 1 : 0, status || 'ready']
pub, pub ? 'public' : 'unlisted', auto_generated ? 1 : 0, status || 'ready']
);
);
+ _syncObject('clip', r.lastInsertRowid);
+ return r;
}
}
function getClipById(id, appId = null) {
function getClipById(id, appId = null) {
@@ -423,7 +469,9 @@ function countClips(appId, filters = {}) {
@@ -423,7 +469,9 @@ function countClips(appId, filters = {}) {
function setClipVisibility(clipId, visibility) {
function setClipVisibility(clipId, visibility) {
const vis = _normVisibility(visibility);
const vis = _normVisibility(visibility);
- return run('UPDATE clips SET visibility = ?, is_public = ? WHERE id = ?', [vis, vis === 'public' ? 1 : 0, clipId]);
+ const r = run('UPDATE clips SET visibility = ?, is_public = ? WHERE id = ?', [vis, vis === 'public' ? 1 : 0, clipId]);
+ _syncObject('clip', clipId);
+ return r;
}
}
function findDuplicateClip({ appId, streamId = null, vodId = null, startTime = 0, endTime = 0, startWindow = 8, endWindow = 10, createdSinceMinutes = 10 }) {
function findDuplicateClip({ appId, streamId = null, vodId = null, startTime = 0, endTime = 0, startWindow = 8, endWindow = 10, createdSinceMinutes = 10 }) {
@@ -564,11 +612,13 @@ function getRecentPasteCommentsByIp(ip, seconds = 10) {
@@ -564,11 +612,13 @@ function getRecentPasteCommentsByIp(ip, seconds = 10) {
// ── File helpers ─────────────────────────────────────────────
// ── File helpers ─────────────────────────────────────────────
function createFile({ key, app_id, user_id, original_name, size, mime, sha256 }) {
function createFile({ key, app_id, user_id, original_name, size, mime, sha256 }) {
- return run(
+ const r = run(
`INSERT INTO files (key, app_id, user_id, original_name, size, mime, sha256)
`INSERT INTO files (key, app_id, user_id, original_name, size, mime, sha256)
VALUES (?, ?, ?, ?, ?, ?, ?)`,
VALUES (?, ?, ?, ?, ?, ?, ?)`,
[key, app_id, user_id || null, original_name || null, size || 0, mime || 'application/octet-stream', sha256 || null]
[key, app_id, user_id || null, original_name || null, size || 0, mime || 'application/octet-stream', sha256 || null]
);
);
+ _syncObject('file', key);
+ return r;
}
}
function getFileByKey(key, appId = null) {
function getFileByKey(key, appId = null) {
@@ -651,7 +701,7 @@ function importLegacyRows(table, rows, appId = 'live') {
@@ -651,7 +701,7 @@ function importLegacyRows(table, rows, appId = 'live') {
}
}
module.exports = {
module.exports = {
- getDb, run, get, all, close,
+ getDb, run, get, all, close, syncObject: _syncObject,
getSetting, setSetting,
getSetting, setSetting,
// apps
// apps
hashApiKey, getApp, listApps, upsertApp, appAllowedOrigins,
hashApiKey, getApp, listApps, upsertApp, appAllowedOrigins,
@@ -53,6 +53,7 @@ CREATE TABLE IF NOT EXISTS vods (
@@ -53,6 +53,7 @@ CREATE TABLE IF NOT EXISTS vods (
ai_transcript TEXT,
ai_transcript TEXT,
ai_analyzed_at DATETIME,
ai_analyzed_at DATETIME,
meta_json TEXT DEFAULT '{}',
meta_json TEXT DEFAULT '{}',
+ object_id TEXT, -- media_objects.id (Wave 4 object model)
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
);
CREATE INDEX IF NOT EXISTS idx_vods_app ON vods(app_id, created_at DESC);
CREATE INDEX IF NOT EXISTS idx_vods_app ON vods(app_id, created_at DESC);
@@ -84,6 +85,7 @@ CREATE TABLE IF NOT EXISTS clips (
@@ -84,6 +85,7 @@ CREATE TABLE IF NOT EXISTS clips (
ai_overview TEXT,
ai_overview TEXT,
ai_transcript TEXT,
ai_transcript TEXT,
ai_analyzed_at DATETIME,
ai_analyzed_at DATETIME,
+ object_id TEXT, -- media_objects.id
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
);
CREATE INDEX IF NOT EXISTS idx_clips_app ON clips(app_id, created_at DESC);
CREATE INDEX IF NOT EXISTS idx_clips_app ON clips(app_id, created_at DESC);
@@ -126,6 +128,7 @@ CREATE TABLE IF NOT EXISTS pastes (
@@ -126,6 +128,7 @@ CREATE TABLE IF NOT EXISTS pastes (
ai_summary TEXT, -- kept for import; Live owns AI generation
ai_summary TEXT, -- kept for import; Live owns AI generation
ai_tags TEXT,
ai_tags TEXT,
ai_analyzed_at DATETIME,
ai_analyzed_at DATETIME,
+ object_id TEXT, -- media_objects.id of the screenshot/avatar bytes (text pastes have none)
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
updated_at DATETIME DEFAULT CURRENT_TIMESTAMP,
updated_at DATETIME DEFAULT CURRENT_TIMESTAMP,
FOREIGN KEY (forked_from) REFERENCES pastes(id) ON DELETE SET NULL
FOREIGN KEY (forked_from) REFERENCES pastes(id) ON DELETE SET NULL
@@ -169,6 +172,7 @@ CREATE TABLE IF NOT EXISTS files (
@@ -169,6 +172,7 @@ CREATE TABLE IF NOT EXISTS files (
size INTEGER NOT NULL DEFAULT 0,
size INTEGER NOT NULL DEFAULT 0,
mime TEXT DEFAULT 'application/octet-stream',
mime TEXT DEFAULT 'application/octet-stream',
sha256 TEXT,
sha256 TEXT,
+ object_id TEXT, -- media_objects.id
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
);
CREATE INDEX IF NOT EXISTS idx_files_app ON files(app_id, created_at DESC);
CREATE INDEX IF NOT EXISTS idx_files_app ON files(app_id, created_at DESC);
@@ -202,3 +206,110 @@ CREATE TABLE IF NOT EXISTS assets (
@@ -202,3 +206,110 @@ CREATE TABLE IF NOT EXISTS assets (
);
);
CREATE INDEX IF NOT EXISTS idx_assets_app_kind ON assets(app_id, kind, created_at DESC);
CREATE INDEX IF NOT EXISTS idx_assets_app_kind ON assets(app_id, kind, created_at DESC);
CREATE UNIQUE INDEX IF NOT EXISTS idx_assets_identity ON assets(app_id, kind, name, channel_username);
CREATE UNIQUE INDEX IF NOT EXISTS idx_assets_identity ON assets(app_id, kind, name, channel_username);
+
+-- ── Canonical object model (roadmap Wave 4; docs/object-model.md) ──
+-- Every stored byte-blob is a media_object. The vods/clips/files/pastes rows
+-- above are typed projections over it (their object_id column), so the old
+-- APIs and URLs keep working while new code speaks in med_<ULID> ids.
+CREATE TABLE IF NOT EXISTS media_objects (
+ id TEXT PRIMARY KEY, -- med_<ULID> (time-sortable)
+ app_id TEXT NOT NULL, -- tenant
+ namespace TEXT NOT NULL, -- capability namespace (= app_id today; projects later)
+ kind TEXT NOT NULL CHECK(kind IN ('vod', 'clip', 'file', 'thumbnail', 'screenshot', 'avatar', 'asset')),
+ owner_subject TEXT, -- canonical subject (usr_<ULID>) when known
+ owner_app TEXT, -- legacy owner: the app whose user-id space owner_user_id is in
+ owner_user_id INTEGER,
+ visibility TEXT NOT NULL DEFAULT 'private' CHECK(visibility IN ('public', 'unlisted', 'private')),
+ lifecycle_status TEXT NOT NULL DEFAULT 'uploading' CHECK(lifecycle_status IN ('uploading', 'ready', 'failed', 'archived', 'deleted')),
+ mime_type TEXT,
+ size_bytes INTEGER NOT NULL DEFAULT 0,
+ content_hash TEXT, -- sha256 hex
+ canonical_provider TEXT, -- local | b2 | r2
+ canonical_key TEXT,
+ legacy_ref TEXT UNIQUE, -- legacy:<app>:<kind>:<id> for projected rows; NULL = native v2 object
+ metadata TEXT NOT NULL DEFAULT '{}',
+ created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
+ updated_at DATETIME DEFAULT CURRENT_TIMESTAMP,
+ deleted_at DATETIME
+);
+CREATE INDEX IF NOT EXISTS idx_media_objects_app ON media_objects(app_id, id);
+CREATE INDEX IF NOT EXISTS idx_media_objects_kind ON media_objects(app_id, kind);
+CREATE INDEX IF NOT EXISTS idx_media_objects_owner ON media_objects(owner_subject);
+CREATE INDEX IF NOT EXISTS idx_media_objects_lifecycle ON media_objects(lifecycle_status);
+
+-- Where the bytes are. One row per provider copy; the canonical one is named on the object.
+CREATE TABLE IF NOT EXISTS media_locations (
+ id INTEGER PRIMARY KEY AUTOINCREMENT,
+ object_id TEXT NOT NULL,
+ provider TEXT NOT NULL CHECK(provider IN ('local', 'b2', 'r2')),
+ bucket TEXT,
+ key TEXT NOT NULL, -- absolute path for local, object key for b2/r2
+ storage_class TEXT, -- hot (local) | cold (b2 canonical) | cache (r2)
+ state TEXT NOT NULL DEFAULT 'pending' CHECK(state IN ('present', 'missing', 'pending', 'corrupt')),
+ checksum TEXT,
+ size_bytes INTEGER,
+ verified_at DATETIME,
+ created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
+ updated_at DATETIME DEFAULT CURRENT_TIMESTAMP,
+ UNIQUE(object_id, provider)
+);
+
+CREATE TABLE IF NOT EXISTS media_relationships (
+ id INTEGER PRIMARY KEY AUTOINCREMENT,
+ from_object_id TEXT NOT NULL,
+ relation TEXT NOT NULL, -- clip_of | thumbnail_of | derived_from | screenshot_of
+ to_object_id TEXT NOT NULL,
+ metadata TEXT NOT NULL DEFAULT '{}',
+ created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
+ UNIQUE(from_object_id, relation, to_object_id)
+);
+CREATE INDEX IF NOT EXISTS idx_media_relationships_to ON media_relationships(to_object_id, relation);
+
+CREATE TABLE IF NOT EXISTS media_variants (
+ id INTEGER PRIMARY KEY AUTOINCREMENT,
+ object_id TEXT NOT NULL, -- the source object
+ variant_name TEXT NOT NULL, -- thumbnail | 720p | waveform | …
+ derived_object_id TEXT NOT NULL,
+ recipe TEXT, -- how it was made (name@version)
+ created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
+ UNIQUE(object_id, variant_name)
+);
+
+-- Generic derivative/maintenance jobs (schema only in this pass — no worker consumes it yet).
+CREATE TABLE IF NOT EXISTS media_jobs (
+ id INTEGER PRIMARY KEY AUTOINCREMENT,
+ object_id TEXT,
+ job_type TEXT NOT NULL,
+ status TEXT NOT NULL DEFAULT 'queued' CHECK(status IN ('queued', 'running', 'done', 'failed', 'cancelled')),
+ attempts INTEGER NOT NULL DEFAULT 0,
+ checkpoint TEXT,
+ error TEXT,
+ created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
+ updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
+);
+CREATE INDEX IF NOT EXISTS idx_media_jobs_status ON media_jobs(status, job_type);
+
+-- Retention holds: an object with an unreleased hold cannot be deleted or moved between tiers.
+CREATE TABLE IF NOT EXISTS media_holds (
+ id INTEGER PRIMARY KEY AUTOINCREMENT,
+ object_id TEXT NOT NULL,
+ kind TEXT NOT NULL CHECK(kind IN ('moderation', 'dmca', 'creator_pin', 'admin', 'evidence')),
+ reason TEXT NOT NULL DEFAULT '',
+ created_by TEXT,
+ created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
+ released_at DATETIME,
+ released_by TEXT
+);
+CREATE INDEX IF NOT EXISTS idx_media_holds_object ON media_holds(object_id, released_at);
+
+-- Public playback objects above the size policy (MEDIA_PUBLIC_OBJECT_*_MB). One row per object.
+CREATE TABLE IF NOT EXISTS media_invariant_violations (
+ id INTEGER PRIMARY KEY AUTOINCREMENT,
+ object_id TEXT NOT NULL UNIQUE,
+ level TEXT NOT NULL CHECK(level IN ('warn', 'violation')),
+ size_bytes INTEGER NOT NULL,
+ threshold_bytes INTEGER NOT NULL,
+ detected_at DATETIME DEFAULT CURRENT_TIMESTAMP,
+ last_seen_at DATETIME DEFAULT CURRENT_TIMESTAMP,
+ resolved_at DATETIME
+);
@@ -171,6 +171,7 @@ router.delete('/:key', tenantAuth({ allowUser: true }), (req, res) => {
@@ -171,6 +171,7 @@ router.delete('/:key', tenantAuth({ allowUser: true }), (req, res) => {
if (req.authType === 'user' && !(req.userId != null && row.user_id === req.userId)) {
if (req.authType === 'user' && !(req.userId != null && row.user_id === req.userId)) {
return res.status(403).json({ error: 'Not authorized to delete this file' });
return res.status(403).json({ error: 'Not authorized to delete this file' });
}
}
+ if (require('../objects/model').isHeldRow(row)) return res.status(409).json({ error: 'File is under a retention hold', code: 'media.object.held' });
const filePath = filePathForKey(row);
const filePath = filePathForKey(row);
try { if (fs.existsSync(filePath)) fs.unlinkSync(filePath); } catch { /* */ }
try { if (fs.existsSync(filePath)) fs.unlinkSync(filePath); } catch { /* */ }
@@ -16,7 +16,7 @@ const auth = require('./auth');
@@ -16,7 +16,7 @@ const auth = require('./auth');
// ── Bootstrap ────────────────────────────────────────────────
// ── Bootstrap ────────────────────────────────────────────────
-for (const dir of [config.vod.path, config.vod.clipsPath, config.pastes.path, config.thumbnails.path, config.files.path]) {
+for (const dir of [config.vod.path, config.vod.clipsPath, config.pastes.path, config.thumbnails.path, config.files.path, config.objects.path]) {
if (!fs.existsSync(dir)) fs.mkdirSync(dir, { recursive: true });
if (!fs.existsSync(dir)) fs.mkdirSync(dir, { recursive: true });
}
}
@@ -27,6 +27,8 @@ auth.startJwksRefresh();
@@ -27,6 +27,8 @@ auth.startJwksRefresh();
const app = express();
const app = express();
app.set('trust proxy', true);
app.set('trust proxy', true);
+// Object API v2 content uploads read the raw request body, so they sit ahead of the body parsers.
+app.put('/api/v2/:app/objects/:id/content', ...require('./objects/routes').contentHandlers);
app.use(express.json({ limit: '2mb' }));
app.use(express.json({ limit: '2mb' }));
// ── Visitor sign-in (OAuth client `media` on the Network; same module as Community/Tools) ──
// ── Visitor sign-in (OAuth client `media` on the Network; same module as Community/Tools) ──
@@ -66,7 +68,7 @@ app.use('/auth', userAuth.createAuthRoutes(userAuthConfig, userAuth.createAuthCl
@@ -66,7 +68,7 @@ app.use('/auth', userAuth.createAuthRoutes(userAuthConfig, userAuth.createAuthCl
app.use(express.urlencoded({ extended: true, limit: '2mb' }));
app.use(express.urlencoded({ extended: true, limit: '2mb' }));
// CORS preflight for tenant API routes (per-app allow-list; auth-less OPTIONS).
// CORS preflight for tenant API routes (per-app allow-list; auth-less OPTIONS).
-app.options('/api/v1/:app/*', (req, res) => {
+app.options(['/api/v1/:app/*', '/api/v2/:app/*'], (req, res) => {
const origin = req.headers.origin;
const origin = req.headers.origin;
const appRow = db.getApp(String(req.params.app || ''));
const appRow = db.getApp(String(req.params.app || ''));
if (origin && appRow && db.appAllowedOrigins(appRow).includes(origin)) {
if (origin && appRow && db.appAllowedOrigins(appRow).includes(origin)) {
@@ -122,6 +124,8 @@ app.use('/api/v1/:app/files', require('./files/routes'));
@@ -122,6 +124,8 @@ app.use('/api/v1/:app/files', require('./files/routes'));
app.use('/api/v1/:app/thumbnails', require('./thumbnails/routes'));
app.use('/api/v1/:app/thumbnails', require('./thumbnails/routes'));
app.use('/api/v1/:app/assets', require('./assets/routes'));
app.use('/api/v1/:app/assets', require('./assets/routes'));
app.use('/api/v1/:app/admin/storage', require('./admin/routes'));
app.use('/api/v1/:app/admin/storage', require('./admin/routes'));
+app.use('/api/v2/:app/objects', require('./objects/routes')); // canonical object API (docs/object-model.md)
+app.use('/o', require('./objects/routes').publicRouter); // object bytes (public, or signed)
app.use('/', require('./public/routes')); // /v /c /p /t /f
app.use('/', require('./public/routes')); // /v /c /p /t /f
// eslint-disable-next-line no-unused-vars
// eslint-disable-next-line no-unused-vars
@@ -159,6 +163,18 @@ every(5 * 60 * 1000, () => {
@@ -159,6 +163,18 @@ every(5 * 60 * 1000, () => {
});
});
every(60 * 1000, () => recorder.checkDisk()); // disk guardian
every(60 * 1000, () => recorder.checkDisk()); // disk guardian
every(60 * 60 * 1000, () => thumbService.cleanupOldThumbnails()); // stale live thumbs
every(60 * 60 * 1000, () => thumbService.cleanupOldThumbnails()); // stale live thumbs
+// Object model: purge native objects whose soft-delete retention has passed (held ones are kept).
+every(60 * 60 * 1000, () => {
+ try { const n = require('./objects/model').purgeExpired(); if (n) console.log(`[Objects] Purged ${n} expired deleted object(s)`); } catch (err) { console.warn('[Objects] purge:', err.message); }
+});
+// Project rows that have no media_object yet (first boot after the upgrade: all of them).
+setTimeout(() => {
+ try {
+ const bf = require('./objects/backfill');
+ const r = bf.backfill({ onlyMissing: true });
+ if (r.totals.created || r.totals.updated || r.errors.length) console.log(`[Objects] Backfill: ${bf.summarize(r)}`);
+ } catch (err) { console.warn('[Objects] Backfill failed:', err.message); }
+}, 15 * 1000).unref?.();
// Recover from an unclean shutdown: rows stuck in is_recording with no live
// Recover from an unclean shutdown: rows stuck in is_recording with no live
// ffmpeg are finalized from whatever hit the disk.
// ffmpeg are finalized from whatever hit the disk.
new file mode 100644
new file mode 100644
@@ -0,0 +1,110 @@
@@ -0,0 +1,110 @@
+/**
+ * OpenVibe.Media — object backfill
+ *
+ * One media_object (+ locations, relationships, thumbnail variant) per existing
+ * vod, clip, file, screenshot/avatar paste and vod/clip thumbnail, derived by the
+ * same sync functions the write hooks use. Idempotent: objects are keyed by their
+ * legacy ref, so a re-run updates in place and creates nothing new.
+ *
+ * Never touches bytes and never calls B2/R2: local copies are 'present' when the
+ * file exists on disk and 'missing' when it does not; remote copies stay 'pending'
+ * until `scripts/reconcile-objects.js --verify` HEADs them.
+ *
+ * Runs in one transaction; dryRun rolls it back and only reports.
+ */
+'use strict';
+
+const db = require('../db/database');
+const model = require('./model');
+
+const SETTING_KEY = 'objects.backfill.last_report';
+
+function emptyCounts() {
+ return { seen: 0, created: 0, updated: 0, skipped: 0 };
+}
+
+function backfill({ dryRun = false, onlyMissing = false } = {}) {
+ const report = {
+ started_at: new Date().toISOString(), dry_run: !!dryRun, only_missing: !!onlyMissing,
+ counts: { vod: emptyCounts(), clip: emptyCounts(), file: emptyCounts(), screenshot: emptyCounts(), avatar: emptyCounts(), thumbnail: emptyCounts() },
+ locations: { present: 0, missing: 0, pending: 0 },
+ skipped: [],
+ errors: [],
+ };
+ const where = onlyMissing ? ' WHERE object_id IS NULL' : '';
+
+ const tally = (kind, table, id, r) => {
+ const c = report.counts[kind];
+ c.seen++;
+ if (!r) return;
+ if (r.skipped) { c.skipped++; report.skipped.push({ table, id, reason: r.skipped }); return; }
+ c[r.created ? 'created' : 'updated']++;
+ for (const s of r.locations || []) if (s in report.locations) report.locations[s]++;
+ };
+ const thumb = (table, id, r) => {
+ if (!r || !r.thumbnail) return;
+ tally('thumbnail', `${table}.thumbnail_url`, id, r.thumbnail);
+ };
+ const d = db.getDb();
+ // One savepoint per row: a row that fails leaves nothing half-written behind.
+ const each = (table, sql, fn) => {
+ const one = d.transaction(fn);
+ for (const row of db.all(sql)) {
+ try { one(row); } catch (err) { report.errors.push({ table, id: row.id ?? row.key, error: err.message }); }
+ }
+ };
+
+ d.exec('BEGIN');
+ try {
+ // VODs before clips, so clip_of can point at the vod's object.
+ each('vods', `SELECT * FROM vods${where} ORDER BY id`, (row) => {
+ const r = model.syncVod(row);
+ tally('vod', 'vods', row.id, r);
+ thumb('vods', row.id, r);
+ });
+ each('clips', `SELECT * FROM clips${where} ORDER BY id`, (row) => {
+ const r = model.syncClip(row);
+ tally('clip', 'clips', row.id, r);
+ thumb('clips', row.id, r);
+ });
+ each('files', `SELECT * FROM files${where} ORDER BY created_at, key`, (row) => {
+ tally('file', 'files', row.key, model.syncFile(row));
+ });
+ each('pastes', `SELECT * FROM pastes${where ? where + ' AND' : ' WHERE'} type = 'screenshot' ORDER BY id`, (row) => {
+ const r = model.syncPaste(row);
+ const kind = (r && r.kind) || (String(row.metadata || '').includes('"kind":"avatar"') ? 'avatar' : 'screenshot');
+ tally(kind, 'pastes', row.id, r);
+ });
+ if (dryRun) d.exec('ROLLBACK'); else d.exec('COMMIT');
+ } catch (err) {
+ try { d.exec('ROLLBACK'); } catch { /* already rolled back */ }
+ throw err;
+ }
+
+ report.finished_at = new Date().toISOString();
+ report.totals = Object.values(report.counts).reduce((t, c) => {
+ for (const k of Object.keys(t)) t[k] += c[k];
+ return t;
+ }, emptyCounts());
+ if (!dryRun && (!onlyMissing || report.totals.created || report.totals.updated)) {
+ // Keep the latest report (skipped rows capped) for operators: media_settings.objects.backfill.last_report.
+ const stored = { ...report, skipped: report.skipped.slice(0, 500), skipped_truncated: report.skipped.length > 500 };
+ db.run(`INSERT INTO media_settings (key, value, description, type) VALUES (?, ?, 'Last object backfill report', 'json')
+ ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = CURRENT_TIMESTAMP`, [SETTING_KEY, JSON.stringify(stored)]);
+ }
+ return report;
+}
+
+function lastReport() {
+ const row = db.get('SELECT value FROM media_settings WHERE key = ?', [SETTING_KEY]);
+ return row ? model.parseJson(row.value, null) : null;
+}
+
+function summarize(report) {
+ const parts = Object.entries(report.counts).filter(([, c]) => c.seen)
+ .map(([k, c]) => `${k} ${c.created}+/${c.updated}~/${c.skipped} skipped`);
+ return `${report.dry_run ? '[dry run] ' : ''}${parts.join(', ') || 'nothing to do'}; locations present ${report.locations.present}, `
+ + `missing ${report.locations.missing}, pending ${report.locations.pending}${report.errors.length ? `; ${report.errors.length} error(s)` : ''}`;
+}
+
+module.exports = { backfill, lastReport, summarize, SETTING_KEY };
new file mode 100644
new file mode 100644
@@ -0,0 +1,94 @@
@@ -0,0 +1,94 @@
+/**
+ * OpenVibe.Media — public object-size invariant (roadmap W4 deliverable 5)
+ *
+ * No public playback object above MEDIA_PUBLIC_OBJECT_MAX_MB (default 500),
+ * target MEDIA_PUBLIC_OBJECT_TARGET_MB (256), warn at MEDIA_PUBLIC_OBJECT_WARN_MB
+ * (384). Policy, not a constant: the thresholds are configuration, the
+ * validator is pure, and violations are recorded in media_invariant_violations
+ * (one row per object, resolved when it drops back under). This pass reports
+ * and refuses new oversized public uploads; it never re-encodes.
+ */
+'use strict';
+
+const db = require('../db/database');
+const config = require('../config');
+
+const MB = 1024 * 1024;
+const PLAYBACK_KINDS = ['vod', 'clip'];
+
+function thresholds(cfg = config.objects) {
+ return { targetBytes: cfg.publicTargetMb * MB, warnBytes: cfg.publicWarnMb * MB, maxBytes: cfg.publicMaxMb * MB };
+}
+
+/** ok ≤ target < above_target ≤ warn < warn-level ≤ max < violation */
+function classify(sizeBytes, t = thresholds()) {
+ const n = Number(sizeBytes) || 0;
+ if (n > t.maxBytes) return 'violation';
+ if (n > t.warnBytes) return 'warn';
+ if (n > t.targetBytes) return 'above_target';
+ return 'ok';
+}
+
+/** Playback bytes anyone with the link can fetch: ready vods/clips that are public or unlisted. */
+function isPublicPlayback(obj) {
+ return !!obj && PLAYBACK_KINDS.includes(obj.kind) && obj.visibility !== 'private' && obj.lifecycle_status === 'ready';
+}
+
+/** Would an object of this kind/visibility/size break the invariant (a public playback object above max)? */
+function wouldViolate({ kind, visibility, size_bytes }, t = thresholds()) {
+ return PLAYBACK_KINDS.includes(kind) && visibility !== 'private' && classify(size_bytes, t) === 'violation';
+}
+
+function validate(obj, t = thresholds()) {
+ const level = classify(obj.size_bytes, t);
+ const threshold = level === 'violation' ? t.maxBytes : level === 'warn' ? t.warnBytes : level === 'above_target' ? t.targetBytes : null;
+ return { object_id: obj.id, kind: obj.kind, visibility: obj.visibility, size_bytes: Number(obj.size_bytes) || 0,
+ level, threshold_bytes: threshold, public_playback: isPublicPlayback(obj) };
+}
+
+/** Upsert (warn/violation) or resolve this object's violation row. Returns the level recorded, or null. */
+function record(obj, t = thresholds()) {
+ if (!obj) return null;
+ const v = validate(obj, t);
+ const open = db.get('SELECT * FROM media_invariant_violations WHERE object_id = ?', [obj.id]);
+ if (v.public_playback && (v.level === 'warn' || v.level === 'violation')) {
+ if (open) {
+ db.run(`UPDATE media_invariant_violations SET level = ?, size_bytes = ?, threshold_bytes = ?, last_seen_at = CURRENT_TIMESTAMP,
+ resolved_at = NULL, detected_at = CASE WHEN resolved_at IS NULL THEN detected_at ELSE CURRENT_TIMESTAMP END
+ WHERE id = ?`, [v.level, v.size_bytes, v.threshold_bytes, open.id]);
+ } else {
+ db.run('INSERT INTO media_invariant_violations (object_id, level, size_bytes, threshold_bytes) VALUES (?, ?, ?, ?)',
+ [obj.id, v.level, v.size_bytes, v.threshold_bytes]);
+ }
+ return v.level;
+ }
+ if (open && !open.resolved_at) db.run('UPDATE media_invariant_violations SET resolved_at = CURRENT_TIMESTAMP WHERE id = ?', [open.id]);
+ return null;
+}
+
+/**
+ * Scan every public playback object. Returns { thresholds, counts, objects } where
+ * objects lists everything above target (largest first). record=false is a dry run.
+ */
+function scan({ record: write = true, appId = null } = {}) {
+ const t = thresholds();
+ const rows = db.all(`SELECT * FROM media_objects WHERE kind IN ('vod', 'clip') AND visibility != 'private' AND lifecycle_status = 'ready'
+ ${appId ? 'AND app_id = ?' : ''} ORDER BY size_bytes DESC`, appId ? [appId] : []);
+ const counts = { ok: 0, above_target: 0, warn: 0, violation: 0 };
+ const objects = [];
+ for (const obj of rows) {
+ const v = validate(obj, t);
+ counts[v.level]++;
+ if (v.level !== 'ok') objects.push({ ...v, app_id: obj.app_id, legacy_ref: obj.legacy_ref });
+ if (write) record(obj, t);
+ }
+ // Objects that stopped being public playback (made private, deleted) resolve too.
+ if (write) {
+ const stale = db.all(`SELECT o.*, v.id AS violation_id FROM media_invariant_violations v LEFT JOIN media_objects o ON o.id = v.object_id
+ WHERE v.resolved_at IS NULL`);
+ for (const s of stale) if (!isPublicPlayback(s)) db.run('UPDATE media_invariant_violations SET resolved_at = CURRENT_TIMESTAMP WHERE id = ?', [s.violation_id]);
+ }
+ return { thresholds: t, counts, objects };
+}
+
+module.exports = { thresholds, classify, isPublicPlayback, wouldViolate, validate, record, scan, MB };
new file mode 100644
new file mode 100644
@@ -0,0 +1,550 @@
@@ -0,0 +1,550 @@
+/**
+ * OpenVibe.Media — canonical object model (roadmap Wave 4; docs/object-model.md)
+ *
+ * Every stored blob is a media_object (id med_<ULID>) with one media_locations
+ * row per provider copy. The inherited vods/clips/files/pastes rows are typed
+ * projections over objects: each carries object_id, and the sync*() functions
+ * below derive the object + locations from the row as it is right now. The same
+ * functions back the one-off backfill and the write hooks on the old APIs, so
+ * the model cannot drift between "imported" and "written since".
+ *
+ * Location states: present (verified: the local file exists / a HEAD answered),
+ * missing (verified absent), pending (believed there, not yet verified — every
+ * remote copy until reconciliation --verify), corrupt (size/hash mismatch).
+ */
+'use strict';
+
+const fs = require('fs');
+const path = require('path');
+const { ids } = require('openvibe-contracts');
+const db = require('../db/database');
+const config = require('../config');
+
+const KINDS = ['vod', 'clip', 'file', 'thumbnail', 'screenshot', 'avatar', 'asset'];
+const VISIBILITIES = ['public', 'unlisted', 'private'];
+const LIFECYCLES = ['uploading', 'ready', 'failed', 'archived', 'deleted'];
+const HOLD_KINDS = ['moderation', 'dmca', 'creator_pin', 'admin', 'evidence'];
+const STORAGE_CLASS = { local: 'hot', b2: 'cold', r2: 'cache' };
+
+const MIME_BY_EXT = {
+ '.webm': 'video/webm', '.mp4': 'video/mp4', '.mkv': 'video/x-matroska', '.mov': 'video/quicktime', '.ogg': 'video/ogg',
+ '.jpg': 'image/jpeg', '.jpeg': 'image/jpeg', '.png': 'image/png', '.webp': 'image/webp', '.gif': 'image/gif',
+ '.mp3': 'audio/mpeg', '.wav': 'audio/wav', '.txt': 'text/plain', '.json': 'application/json', '.pdf': 'application/pdf',
+};
+
+class HeldError extends Error {
+ constructor(objectId) { super('media object is under a retention hold'); this.code = 'media.object.held'; this.objectId = objectId; }
+}
+
+// ── Small helpers ────────────────────────────────────────────
+
+function mimeFor(p, fallback = 'application/octet-stream') {
+ return MIME_BY_EXT[path.extname(String(p || '')).toLowerCase()] || fallback;
+}
+
+function parseJson(s, fallback) {
+ if (s == null || s === '') return fallback;
+ try { return JSON.parse(s); } catch { return fallback; }
+}
+
+/** SQLite 'YYYY-MM-DD HH:MM:SS' (UTC) → epoch ms, so backfilled ids sort by original creation time. */
+function toMs(createdAt) {
+ const ms = Date.parse(String(createdAt || '').replace(' ', 'T') + (/[zZ]|[+-]\d\d:?\d\d$/.test(String(createdAt || '')) ? '' : 'Z'));
+ return Number.isFinite(ms) ? ms : Date.now();
+}
+
+function statFile(p) {
+ if (!p) return null;
+ try { const st = fs.statSync(p); return st.isFile() ? { size: st.size } : null; } catch { return null; }
+}
+
+function rowVisibility(row) {
+ const v = row.visibility || (row.is_public ? 'public' : 'private');
+ return VISIBILITIES.includes(v) ? v : 'private';
+}
+
+function legacyRef(app, kind, id) {
+ return ids.legacyMediaId(app, kind, id);
+}
+
+function parseLegacyRef(ref) {
+ const m = /^legacy:([^:]+):([^:]+):(.+)$/.exec(String(ref || ''));
+ return m ? { app: m[1], kind: m[2], id: m[3] } : null;
+}
+
+const vodStorage = () => require('../vod/vod-storage');
+
+// ── Reads ────────────────────────────────────────────────────
+
+function getObject(id) {
+ return id ? db.get('SELECT * FROM media_objects WHERE id = ?', [id]) : null;
+}
+
+function getObjectByLegacyRef(ref) {
+ return ref ? db.get('SELECT * FROM media_objects WHERE legacy_ref = ?', [ref]) : null;
+}
+
+/** An object by med_ id or legacy ref, scoped to one tenant (null when it belongs to another). */
+function resolveObject(idOrRef, appId) {
+ const key = String(idOrRef || '');
+ const obj = key.startsWith('legacy:') ? getObjectByLegacyRef(key) : getObject(key);
+ return obj && obj.app_id === appId ? obj : null;
+}
+
+function listLocations(objectId) {
+ return db.all('SELECT * FROM media_locations WHERE object_id = ? ORDER BY id', [objectId]);
+}
+
+function listHolds(objectId, { includeReleased = false } = {}) {
+ return db.all(`SELECT * FROM media_holds WHERE object_id = ?${includeReleased ? '' : ' AND released_at IS NULL'} ORDER BY id`, [objectId]);
+}
+
+function isHeld(objectId) {
+ if (!objectId) return false;
+ return !!db.get('SELECT 1 AS x FROM media_holds WHERE object_id = ? AND released_at IS NULL LIMIT 1', [objectId]);
+}
+
+/** Hold check for an inherited row (vods/clips/files/pastes) — false when it has no object yet. */
+function isHeldRow(row) {
+ try { return !!(row && row.object_id && isHeld(row.object_id)); } catch { return false; }
+}
+
+// ── Writes ───────────────────────────────────────────────────
+
+const OBJECT_COLUMNS = ['app_id', 'namespace', 'kind', 'owner_subject', 'owner_app', 'owner_user_id', 'visibility',
+ 'lifecycle_status', 'mime_type', 'size_bytes', 'content_hash', 'canonical_provider', 'canonical_key', 'legacy_ref', 'metadata', 'deleted_at'];
+
+function createObject(o) {
+ if (!KINDS.includes(o.kind)) throw new Error(`unknown kind ${o.kind}`);
+ const id = ids.newId('media', o.createdMs);
+ db.run(`INSERT INTO media_objects (id, app_id, namespace, kind, owner_subject, owner_app, owner_user_id, visibility,
+ lifecycle_status, mime_type, size_bytes, content_hash, canonical_provider, canonical_key, legacy_ref, metadata, created_at)
+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, COALESCE(?, CURRENT_TIMESTAMP))`,
+ [id, o.app_id, o.namespace || o.app_id, o.kind, o.owner_subject || null, o.owner_app || null, o.owner_user_id ?? null,
+ VISIBILITIES.includes(o.visibility) ? o.visibility : 'private', LIFECYCLES.includes(o.lifecycle_status) ? o.lifecycle_status : 'uploading',
+ o.mime_type || null, Number(o.size_bytes) || 0, o.content_hash || null, o.canonical_provider || null, o.canonical_key || null,
+ o.legacy_ref || null, JSON.stringify(o.metadata || {}), o.created_at || null]);
+ return id;
+}
+
+function updateObject(id, fields) {
+ const sets = [], params = [];
+ for (const [k, v] of Object.entries(fields)) {
+ if (!OBJECT_COLUMNS.includes(k)) continue;
+ sets.push(`${k} = ?`);
+ params.push(k === 'metadata' && typeof v !== 'string' ? JSON.stringify(v || {}) : v);
+ }
+ if (!sets.length) return;
+ params.push(id);
+ db.run(`UPDATE media_objects SET ${sets.join(', ')}, updated_at = CURRENT_TIMESTAMP WHERE id = ?`, params);
+}
+
+/**
+ * Upsert one provider copy. A remote copy re-derived as 'pending' keeps what
+ * reconciliation already verified about the same key (never downgrade knowledge).
+ */
+function upsertLocation(objectId, loc) {
+ const existing = db.get('SELECT * FROM media_locations WHERE object_id = ? AND provider = ?', [objectId, loc.provider]);
+ const verifiedAt = loc.verified ? new Date().toISOString().replace('T', ' ').slice(0, 19) : null;
+ if (!existing) {
+ db.run(`INSERT INTO media_locations (object_id, provider, bucket, key, storage_class, state, checksum, size_bytes, verified_at)
+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,
+ [objectId, loc.provider, loc.bucket || null, loc.key, loc.storage_class || STORAGE_CLASS[loc.provider] || null,
+ loc.state || 'pending', loc.checksum || null, loc.size_bytes ?? null, verifiedAt]);
+ return;
+ }
+ const keep = existing.key === loc.key && (loc.state || 'pending') === 'pending' && existing.state !== 'pending';
+ db.run(`UPDATE media_locations SET bucket = ?, key = ?, storage_class = ?, state = ?, checksum = ?, size_bytes = ?, verified_at = ?,
+ updated_at = CURRENT_TIMESTAMP WHERE id = ?`,
+ [loc.bucket ?? existing.bucket, loc.key, loc.storage_class || existing.storage_class,
+ keep ? existing.state : (loc.state || 'pending'),
+ loc.checksum ?? (existing.key === loc.key ? existing.checksum : null),
+ keep ? existing.size_bytes : (loc.size_bytes ?? null),
+ keep ? existing.verified_at : (verifiedAt || (existing.key === loc.key && loc.state === existing.state ? existing.verified_at : null)),
+ existing.id]);
+}
+
+function setLocationState(locationId, { state, size_bytes, verified = true }) {
+ db.run(`UPDATE media_locations SET state = ?, size_bytes = COALESCE(?, size_bytes), verified_at = ${verified ? 'CURRENT_TIMESTAMP' : 'verified_at'},
+ updated_at = CURRENT_TIMESTAMP WHERE id = ?`, [state, size_bytes ?? null, locationId]);
+}
+
+function setRelationship(fromId, relation, toId, metadata = {}) {
+ db.run(`INSERT INTO media_relationships (from_object_id, relation, to_object_id, metadata) VALUES (?, ?, ?, ?)
+ ON CONFLICT(from_object_id, relation, to_object_id) DO UPDATE SET metadata = excluded.metadata`,
+ [fromId, relation, toId, JSON.stringify(metadata)]);
+}
+
+function setVariant(objectId, variantName, derivedId, recipe = null) {
+ db.run(`INSERT INTO media_variants (object_id, variant_name, derived_object_id, recipe) VALUES (?, ?, ?, ?)
+ ON CONFLICT(object_id, variant_name) DO UPDATE SET derived_object_id = excluded.derived_object_id, recipe = excluded.recipe`,
+ [objectId, variantName, derivedId, recipe]);
+}
+
+function getVariant(objectId, variantName) {
+ return db.get('SELECT * FROM media_variants WHERE object_id = ? AND variant_name = ?', [objectId, variantName]);
+}
+
+// ── Projection core ──────────────────────────────────────────
+
+/**
+ * Create or update the object behind one inherited row and make its locations
+ * exactly the derived set. Returns { id, created }.
+ * p.locations [{ provider, bucket, key, state, size_bytes, checksum }]
+ * p.canonical preferred canonical provider (falls back to the first location)
+ */
+function project(p) {
+ const found = getObjectByLegacyRef(p.legacy_ref) || (p.existingId ? getObject(p.existingId) : null);
+ const canon = p.locations.find(l => l.provider === p.canonical) || p.locations[0] || null;
+ const fields = {
+ app_id: p.app_id, namespace: p.app_id, kind: p.kind,
+ owner_app: p.owner_app || p.app_id, owner_user_id: p.owner_user_id ?? null,
+ visibility: p.visibility, lifecycle_status: p.lifecycle_status,
+ mime_type: p.mime_type || null, size_bytes: Number(p.size_bytes) || 0, content_hash: p.content_hash || null,
+ canonical_provider: canon ? canon.provider : null, canonical_key: canon ? canon.key : null,
+ legacy_ref: p.legacy_ref, metadata: p.metadata || {},
+ };
+ let id, created = false;
+ if (found) {
+ id = found.id;
+ // Knowledge the row doesn't carry survives a re-projection.
+ if (!fields.content_hash) fields.content_hash = found.content_hash;
+ fields.metadata = { ...parseJson(found.metadata, {}), ...fields.metadata };
+ if (found.lifecycle_status === 'deleted') fields.deleted_at = null; // the row exists, so the object does
+ updateObject(id, fields);
+ } else {
+ id = createObject({ ...fields, createdMs: toMs(p.created_at), created_at: p.created_at || null });
+ created = true;
+ }
+ const keep = new Set(p.locations.map(l => l.provider));
+ for (const loc of p.locations) upsertLocation(id, loc);
+ for (const l of listLocations(id)) if (!keep.has(l.provider)) db.run('DELETE FROM media_locations WHERE id = ?', [l.id]);
+ return { id, created };
+}
+
+function localLocation(candidates, { mustExist = false } = {}) {
+ for (const p of candidates.filter(Boolean)) {
+ const st = statFile(p);
+ if (st) return { provider: 'local', key: path.resolve(p), state: 'present', size_bytes: st.size, verified: true };
+ }
+ if (mustExist || !candidates[0]) return null;
+ return { provider: 'local', key: path.resolve(candidates[0]), state: 'missing', size_bytes: null, verified: true };
+}
+
+/** local + B2 canonical + R2 cache copies for a vods/clips row (same columns, same key scheme). */
+function tieredLocations(row, localCandidates) {
+ const vs = vodStorage();
+ const provider = vs.providerOf(row);
+ const remoteKnown = provider !== 'local' || !!row.storage_key;
+ const locs = [];
+ const local = row.file_path ? localLocation(localCandidates, { mustExist: provider !== 'local' }) : null;
+ if (local) locs.push(local);
+ if (remoteKnown && row.file_path) {
+ const key = vs.keyForVod(row);
+ locs.push({ provider: 'b2', bucket: vs.bucketFor('b2'), key, state: 'pending' });
+ if (provider === 'r2') locs.push({ provider: 'r2', bucket: vs.bucketFor('r2'), key, state: 'pending' });
+ }
+ return { locs, canonical: remoteKnown ? 'b2' : 'local' };
+}
+
+function linkRow(table, keyCol, keyVal, objectId) {
+ db.run(`UPDATE ${table} SET object_id = ? WHERE ${keyCol} = ? AND (object_id IS NULL OR object_id != ?)`, [objectId, keyVal, objectId]);
+}
+
+function recordInvariant(objectId) {
+ try { require('./invariant').record(getObject(objectId)); } catch (err) { console.warn('[Objects] invariant record:', err.message); }
+}
+
+// ── Per-kind projections ─────────────────────────────────────
+
+function vodLifecycle(row) {
+ const s = db.vodStatus(row);
+ if (s === 'ready' || s === 'failed') return s;
+ return 'uploading';
+}
+
+function syncVod(idOrRow) {
+ const row = typeof idOrRow === 'object' ? idOrRow : db.get('SELECT * FROM vods WHERE id = ?', [idOrRow]);
+ if (!row) return null;
+ if (row.clips_only) return { skipped: 'clips-only recording (ephemeral, never published)' };
+ const vs = vodStorage();
+ const { locs, canonical } = tieredLocations(row, [row.file_path ? vs.localPathForVod(row) : null, row.file_path]);
+ const local = locs.find(l => l.provider === 'local' && l.state === 'present');
+ const r = project({
+ legacy_ref: legacyRef(row.app_id, 'vod', row.id), existingId: row.object_id,
+ app_id: row.app_id, kind: 'vod', owner_user_id: row.user_id,
+ visibility: rowVisibility(row), lifecycle_status: vodLifecycle(row),
+ mime_type: row.file_path ? mimeFor(row.file_path, 'video/webm') : null,
+ size_bytes: Number(row.file_size) || (local ? local.size_bytes : 0),
+ metadata: { title: row.title || null, duration_seconds: Number(row.duration_seconds) || 0, health_status: row.health_status || null,
+ managed_stream_id: row.managed_stream_id || null, recording: !!row.is_recording },
+ created_at: row.created_at, locations: locs, canonical,
+ });
+ linkRow('vods', 'id', row.id, r.id);
+ recordInvariant(r.id);
+ const thumbnail = syncThumbnail(row, r.id, 'vod');
+ return { ...r, locations: locs.map(l => l.state), thumbnail };
+}
+
+function syncClip(idOrRow) {
+ const row = typeof idOrRow === 'object' ? idOrRow : db.get('SELECT * FROM clips WHERE id = ?', [idOrRow]);
+ if (!row) return null;
+ const clipsDir = path.resolve(config.vod.clipsPath);
+ const { locs, canonical } = tieredLocations(row, [row.file_path || null, row.file_path ? path.join(clipsDir, path.basename(row.file_path)) : null]);
+ const status = row.status || 'ready';
+ const lifecycle = status === 'processing' ? 'uploading' : status === 'failed' ? 'failed' : (row.file_path ? 'ready' : 'failed');
+ const local = locs.find(l => l.provider === 'local' && l.state === 'present');
+ const r = project({
+ legacy_ref: legacyRef(row.app_id, 'clip', row.id), existingId: row.object_id,
+ app_id: row.app_id, kind: 'clip', owner_user_id: row.user_id,
+ visibility: rowVisibility(row), lifecycle_status: lifecycle,
+ mime_type: row.file_path ? mimeFor(row.file_path, 'video/webm') : null,
+ size_bytes: local ? local.size_bytes : 0,
+ metadata: { title: row.title || null, duration_seconds: Number(row.duration_seconds) || 0, start_time: row.start_time, end_time: row.end_time,
+ channel_user_id: row.channel_user_id || null, auto_generated: !!row.auto_generated },
+ created_at: row.created_at, locations: locs, canonical,
+ });
+ linkRow('clips', 'id', row.id, r.id);
+ if (row.vod_id) {
+ const vod = db.get('SELECT object_id FROM vods WHERE id = ?', [row.vod_id]);
+ if (vod && vod.object_id) setRelationship(r.id, 'clip_of', vod.object_id, { start_time: row.start_time, end_time: row.end_time });
+ }
+ recordInvariant(r.id);
+ const thumbnail = syncThumbnail(row, r.id, 'clip');
+ return { ...r, locations: locs.map(l => l.state), thumbnail };
+}
+
+function syncFile(keyOrRow) {
+ const row = typeof keyOrRow === 'object' ? keyOrRow : db.get('SELECT * FROM files WHERE key = ?', [keyOrRow]);
+ if (!row) return null;
+ const local = localLocation([path.join(config.files.path, row.app_id, row.key)]);
+ if (local.state === 'present' && row.sha256) local.checksum = row.sha256;
+ const r = project({
+ legacy_ref: legacyRef(row.app_id, 'file', row.key), existingId: row.object_id,
+ app_id: row.app_id, kind: 'file', owner_user_id: row.user_id,
+ visibility: 'public', // /f/:key serves every file without auth
+ lifecycle_status: 'ready', mime_type: row.mime || 'application/octet-stream',
+ size_bytes: Number(row.size) || 0, content_hash: row.sha256 || null,
+ metadata: { filename: row.original_name || null },
+ created_at: row.created_at, locations: [local], canonical: 'local',
+ });
+ linkRow('files', 'key', row.key, r.id);
+ return { ...r, locations: [local.state] };
+}
+
+/** Screenshot pastes (and avatars, which are stored as screenshot pastes) — the bytes only; text is Community's. */
+function syncPaste(idOrRow) {
+ const row = typeof idOrRow === 'object' ? idOrRow : db.get('SELECT * FROM pastes WHERE id = ?', [idOrRow]);
+ if (!row) return null;
+ if (row.type !== 'screenshot') return { skipped: 'text paste (no bytes)' };
+ if (!row.screenshot_path) return { skipped: 'screenshot paste without a file path' };
+ const meta = parseJson(row.metadata, {}) || {};
+ const kind = meta.kind === 'avatar' ? 'avatar' : 'screenshot';
+ const local = localLocation([row.screenshot_path]);
+ const r = project({
+ legacy_ref: legacyRef(row.app_id, kind === 'avatar' ? 'avatar' : 'paste', row.slug), existingId: row.object_id,
+ app_id: row.app_id, kind, owner_user_id: row.user_id,
+ visibility: VISIBILITIES.includes(row.visibility) ? row.visibility : 'public', lifecycle_status: 'ready',
+ mime_type: meta.mime_type || mimeFor(row.screenshot_path, 'image/png'),
+ size_bytes: local.state === 'present' ? local.size_bytes : (Number(meta.size_bytes) || 0),
+ metadata: { title: row.title || null, slug: row.slug },
+ created_at: row.created_at, locations: [local], canonical: 'local',
+ });
+ linkRow('pastes', 'id', row.id, r.id);
+ return { ...r, kind, locations: [local.state] };
+}
+
+/** Thumbnail file name behind a stored thumbnail_url, or null when it is not one of ours. */
+function thumbFileFromUrl(u) {
+ if (!u) return null;
+ let p = String(u);
+ if (/^https?:\/\//i.test(p)) {
+ if (!p.startsWith(config.publicUrl + '/')) return null;
+ p = p.slice(config.publicUrl.length);
+ }
+ const m = /^\/(?:t|api\/thumbnails)\/([A-Za-z0-9._-]+)$/.exec(p);
+ return m ? m[1] : null;
+}
+
+/**
+ * A vod/clip's thumbnail is one object per parent, updated in place when the
+ * picture is regenerated (live recordings refresh theirs every couple of minutes).
+ */
+function syncThumbnail(parentRow, parentObjectId, parentKind) {
+ if (!parentRow.thumbnail_url) return null;
+ const name = thumbFileFromUrl(parentRow.thumbnail_url);
+ if (!name) return { skipped: 'external thumbnail url' };
+ const local = localLocation([path.join(path.resolve(config.thumbnails.path), name)]);
+ const variant = getVariant(parentObjectId, 'thumbnail');
+ const parentVis = rowVisibility(parentRow);
+ const r = project({
+ legacy_ref: legacyRef(parentRow.app_id, 'thumbnail', name), existingId: variant ? variant.derived_object_id : null,
+ app_id: parentRow.app_id, kind: 'thumbnail', owner_user_id: parentRow.user_id,
+ // /t/<name> is served to anyone holding the name.
+ visibility: parentVis === 'public' ? 'public' : 'unlisted', lifecycle_status: 'ready',
+ mime_type: mimeFor(name, 'image/jpeg'), size_bytes: local.state === 'present' ? local.size_bytes : 0,
+ metadata: { of: parentKind },
+ created_at: parentRow.created_at, locations: [local], canonical: 'local',
+ });
+ setVariant(parentObjectId, 'thumbnail', r.id);
+ setRelationship(r.id, 'thumbnail_of', parentObjectId);
+ return { ...r, locations: [local.state] };
+}
+
+function _syncTx(fn, arg) {
+ return db.getDb().transaction(() => fn(arg))();
+}
+
+/** Re-project one inherited row: kind vod|clip|file|paste, id = row id (file key for files). */
+function sync(kind, id) {
+ const fn = { vod: syncVod, clip: syncClip, file: syncFile, paste: syncPaste }[kind];
+ if (!fn || id == null) return null;
+ return _syncTx(fn, id);
+}
+
+/** Never-throwing variant for write hooks outside database.js. */
+function safeSync(kind, id) {
+ try { return sync(kind, id); } catch (err) { console.warn(`[Objects] sync ${kind} ${id}:`, err.message); return null; }
+}
+
+/**
+ * After a tier move that verified a copy (upload + HEAD, copy + HEAD): re-project,
+ * then mark the verified providers present.
+ */
+function afterTierMove(vodId, verifiedProviders = []) {
+ try {
+ const r = sync('vod', vodId);
+ if (!r || !r.id) return;
+ for (const l of listLocations(r.id)) {
+ if (verifiedProviders.includes(l.provider)) setLocationState(l.id, { state: 'present' });
+ }
+ } catch (err) { console.warn(`[Objects] tier sync vod ${vodId}:`, err.message); }
+}
+
+// ── Holds ────────────────────────────────────────────────────
+
+function placeHold({ object_id, kind, reason = '', created_by = null }) {
+ if (!HOLD_KINDS.includes(kind)) throw new Error(`hold kind must be one of ${HOLD_KINDS.join(', ')}`);
+ const r = db.run('INSERT INTO media_holds (object_id, kind, reason, created_by) VALUES (?, ?, ?, ?)',
+ [object_id, kind, String(reason || '').slice(0, 1000), created_by]);
+ return db.get('SELECT * FROM media_holds WHERE id = ?', [r.lastInsertRowid]);
+}
+
+function releaseHold(holdId, releasedBy = null) {
+ db.run('UPDATE media_holds SET released_at = CURRENT_TIMESTAMP, released_by = ? WHERE id = ? AND released_at IS NULL', [releasedBy, holdId]);
+ return db.get('SELECT * FROM media_holds WHERE id = ?', [holdId]);
+}
+
+// ── Native objects: soft delete, restore, purge, quota ───────
+
+function objectFilePath(obj) {
+ return path.join(config.objects.path, obj.app_id, obj.id);
+}
+
+function softDelete(obj, { by = null } = {}) {
+ if (isHeld(obj.id)) throw new HeldError(obj.id);
+ const md = parseJson(obj.metadata, {});
+ md.pre_delete_status = obj.lifecycle_status;
+ md.retention_until = new Date(Date.now() + config.objects.retentionDays * 864e5).toISOString();
+ if (by) md.deleted_by = by;
+ db.run(`UPDATE media_objects SET lifecycle_status = 'deleted', deleted_at = CURRENT_TIMESTAMP, metadata = ?, updated_at = CURRENT_TIMESTAMP
+ WHERE id = ?`, [JSON.stringify(md), obj.id]);
+ return getObject(obj.id);
+}
+
+function restore(obj) {
+ const md = parseJson(obj.metadata, {});
+ if (md.purged_at) return null;
+ const back = md.pre_delete_status && md.pre_delete_status !== 'deleted' ? md.pre_delete_status : 'ready';
+ delete md.pre_delete_status; delete md.retention_until; delete md.deleted_by;
+ db.run(`UPDATE media_objects SET lifecycle_status = ?, deleted_at = NULL, metadata = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ?`,
+ [back, JSON.stringify(md), obj.id]);
+ return getObject(obj.id);
+}
+
+/**
+ * Remove the bytes of native (v2) objects deleted longer ago than the retention
+ * period. Projected objects are never purged here — their bytes belong to the
+ * inherited pipelines. Held objects are skipped.
+ */
+function purgeExpired({ retentionDays = config.objects.retentionDays } = {}) {
+ const rows = db.all(`SELECT * FROM media_objects WHERE lifecycle_status = 'deleted' AND legacy_ref IS NULL
+ AND deleted_at <= datetime('now', ?)`, [`-${Math.max(0, retentionDays)} days`]);
+ const root = path.resolve(config.objects.path) + path.sep;
+ let purged = 0;
+ for (const obj of rows) {
+ const md = parseJson(obj.metadata, {});
+ if (md.purged_at || isHeld(obj.id)) continue;
+ for (const l of listLocations(obj.id)) {
+ if (l.provider === 'local' && path.resolve(l.key).startsWith(root)) { try { fs.unlinkSync(l.key); } catch { /* already gone */ } }
+ if (l.provider === 'local') db.run('DELETE FROM media_locations WHERE id = ?', [l.id]);
+ }
+ md.purged_at = new Date().toISOString();
+ db.run('UPDATE media_objects SET metadata = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ?', [JSON.stringify(md), obj.id]);
+ purged++;
+ }
+ return purged;
+}
+
+/** Bytes counted against apps.quota_bytes: v1 files + live native objects (reservations included). */
+function usedBytes(appId, excludeId = null) {
+ const native = db.get(`SELECT COALESCE(SUM(size_bytes), 0) AS b FROM media_objects
+ WHERE app_id = ? AND legacy_ref IS NULL AND lifecycle_status IN ('uploading', 'ready') AND id != ?`,
+ [appId, excludeId || '']).b;
+ return db.appFilesBytes(appId) + native;
+}
+
+// ── Public shape ─────────────────────────────────────────────
+
+/** Where a projected object is served today (the inherited public URL), or null. */
+function legacyPublicUrl(obj) {
+ const ref = parseLegacyRef(obj.legacy_ref);
+ if (!ref) return null;
+ const id = encodeURIComponent(ref.id);
+ const p = { vod: `/v/${id}`, clip: `/c/${id}`, file: `/f/${id}`, thumbnail: `/t/${id}`, paste: `/p/${id}/screenshot`, avatar: `/p/${id}/screenshot` }[ref.kind];
+ return p ? config.publicUrl + p : null;
+}
+
+function objectPublic(obj, { locations = true } = {}) {
+ if (!obj) return null;
+ const out = {
+ id: obj.id,
+ media_ref: { media_id: obj.id },
+ legacy_ref: obj.legacy_ref || null,
+ app_id: obj.app_id,
+ namespace: obj.namespace,
+ kind: obj.kind,
+ owner: { subject: obj.owner_subject || null, app: obj.owner_app || null, user_id: obj.owner_user_id ?? null },
+ visibility: obj.visibility,
+ lifecycle_status: obj.lifecycle_status,
+ mime_type: obj.mime_type,
+ size_bytes: obj.size_bytes,
+ content_hash: obj.content_hash || null,
+ metadata: parseJson(obj.metadata, {}),
+ held: isHeld(obj.id),
+ public_url: obj.visibility !== 'private' && obj.lifecycle_status === 'ready'
+ ? (legacyPublicUrl(obj) || `${config.publicUrl}/o/${obj.id}`) : null,
+ created_at: obj.created_at,
+ updated_at: obj.updated_at,
+ deleted_at: obj.deleted_at || null,
+ };
+ // Where the bytes are, never the keys or paths themselves.
+ if (locations) {
+ out.locations = listLocations(obj.id).map(l => ({
+ provider: l.provider, storage_class: l.storage_class, state: l.state, size_bytes: l.size_bytes,
+ verified_at: l.verified_at, canonical: l.provider === obj.canonical_provider,
+ }));
+ }
+ return out;
+}
+
+module.exports = {
+ KINDS, VISIBILITIES, LIFECYCLES, HOLD_KINDS, HeldError,
+ mimeFor, parseJson, parseLegacyRef, thumbFileFromUrl,
+ getObject, getObjectByLegacyRef, resolveObject, listLocations, listHolds, isHeld, isHeldRow,
+ createObject, updateObject, upsertLocation, setLocationState, setRelationship, setVariant, getVariant,
+ syncVod, syncClip, syncFile, syncPaste, sync, safeSync, afterTierMove,
+ placeHold, releaseHold,
+ objectFilePath, softDelete, restore, purgeExpired, usedBytes,
+ legacyPublicUrl, objectPublic,
+};
new file mode 100644
new file mode 100644
@@ -0,0 +1,197 @@
@@ -0,0 +1,197 @@
+/**
+ * OpenVibe.Media — object reconciliation (docs/object-model.md#reconciliation)
+ *
+ * Compares the object model with the bytes. Read-only by default: it checks
+ * local files and the database and reports. With verify=true it also HEADs
+ * every B2/R2 location (through the storage engine's S3 client, or an injected
+ * `head` for tests) and writes what it learned back to media_locations
+ * (state present/missing/corrupt, size, verified_at). Never deletes anything.
+ *
+ * Detects:
+ * no_canonical_location ready object with no row for its canonical copy
+ * canonical_missing_replica_present canonical copy gone/corrupt while another copy is present
+ * no_present_copy ready object with no present copy anywhere
+ * missing_local_file ready object's local location whose file is not on disk
+ * remote_missing ready object's B2/R2 location whose HEAD says 404 (verify)
+ * size_mismatch / hash_mismatch bytes differ from the object's size / sha256
+ * orphan_location location rows whose object does not exist
+ * deleted_publicly_reachable deleted object still served (legacy row, or a thumbnail of it)
+ * missing_projection inherited rows with no object yet (run the backfill)
+ */
+'use strict';
+
+const fs = require('fs');
+const crypto = require('crypto');
+const db = require('../db/database');
+const model = require('./model');
+
+const MB = 1024 * 1024;
+const ISSUES = ['no_canonical_location', 'canonical_missing_replica_present', 'no_present_copy', 'missing_local_file', 'remote_missing',
+ 'size_mismatch', 'hash_mismatch', 'orphan_location', 'deleted_publicly_reachable', 'missing_projection'];
+
+/** Real provider: undefined when the provider is not configured here (cannot verify), null on 404. */
+async function defaultHead(provider, key) {
+ const vs = require('../vod/vod-storage');
+ if (!vs.providerConfigured(provider)) return undefined;
+ return vs.headObject(provider, key);
+}
+
+function sha256File(p) {
+ return new Promise((resolve, reject) => {
+ const h = crypto.createHash('sha256');
+ const s = fs.createReadStream(p);
+ s.on('data', d => h.update(d));
+ s.on('end', () => resolve(h.digest('hex')));
+ s.on('error', reject);
+ });
+}
+
+// Which inherited row serves a projected object, and is it reachable without credentials?
+const PROJECTIONS = [
+ { table: 'vods', open: (r) => (r.visibility || (r.is_public ? 'public' : 'private')) !== 'private' && !r.clips_only },
+ { table: 'clips', open: (r) => (r.visibility || (r.is_public ? 'public' : 'private')) !== 'private' },
+ { table: 'files', open: () => true },
+ { table: 'pastes', open: (r) => r.visibility !== 'private' && !!r.screenshot_path },
+];
+
+async function reconcile({ verify = false, hash = false, head = defaultHead, hashMaxBytes = 512 * MB, appId = null, listLimit = 200 } = {}) {
+ const report = {
+ generated_at: new Date().toISOString(),
+ mode: { verify: !!verify, hash: !!hash, app_id: appId },
+ counts: { objects: 0, locations: 0, local_checked: 0, remote_checked: 0, remote_unverified: 0, remote_unverifiable: 0, locations_updated: 0 },
+ issues: Object.fromEntries(ISSUES.map(k => [k, { count: 0, items: [] }])),
+ };
+ const issue = (k, item) => { const i = report.issues[k]; i.count++; if (i.items.length < listLimit) i.items.push(item); };
+
+ const objects = db.all(`SELECT * FROM media_objects${appId ? ' WHERE app_id = ?' : ''}`, appId ? [appId] : []);
+ const byId = new Map(objects.map(o => [o.id, o]));
+ const locations = appId
+ ? db.all('SELECT l.* FROM media_locations l LEFT JOIN media_objects o ON o.id = l.object_id WHERE o.app_id = ? OR o.id IS NULL', [appId])
+ : db.all('SELECT * FROM media_locations');
+ report.counts.objects = objects.length;
+ report.counts.locations = locations.length;
+
+ const effective = new Map(); // location id → state after this run's checks
+ const byObject = new Map();
+ const write = (loc, state, size) => {
+ if (!verify) return;
+ model.setLocationState(loc.id, { state, size_bytes: size });
+ report.counts.locations_updated++;
+ };
+
+ for (const loc of locations) {
+ const obj = byId.get(loc.object_id);
+ if (!obj) { issue('orphan_location', { location_id: loc.id, object_id: loc.object_id, provider: loc.provider }); continue; }
+ if (!byObject.has(obj.id)) byObject.set(obj.id, []);
+ byObject.get(obj.id).push(loc);
+ // Missing bytes are an issue for ready objects; a recording or upload may not have written yet.
+ const live = obj.lifecycle_status === 'ready';
+ const expected = Number(obj.size_bytes) || 0;
+ const sizeMatters = obj.lifecycle_status === 'ready' && expected > 0;
+
+ if (loc.provider === 'local') {
+ report.counts.local_checked++;
+ let st = null;
+ try { const s = fs.statSync(loc.key); if (s.isFile()) st = s; } catch { /* missing */ }
+ if (!st) {
+ effective.set(loc.id, 'missing');
+ if (live) issue('missing_local_file', { object_id: obj.id, kind: obj.kind, legacy_ref: obj.legacy_ref, path: loc.key });
+ write(loc, 'missing', null);
+ continue;
+ }
+ let state = 'present';
+ if (sizeMatters && st.size !== expected) {
+ issue('size_mismatch', { object_id: obj.id, provider: 'local', expected, actual: st.size, legacy_ref: obj.legacy_ref });
+ }
+ if (hash && obj.content_hash && st.size <= hashMaxBytes) {
+ const actual = await sha256File(loc.key).catch(() => null);
+ if (actual && actual !== obj.content_hash) {
+ state = 'corrupt';
+ issue('hash_mismatch', { object_id: obj.id, provider: 'local', expected: obj.content_hash, actual });
+ }
+ }
+ effective.set(loc.id, state);
+ write(loc, state, st.size);
+ continue;
+ }
+
+ // b2 / r2
+ if (!verify) {
+ effective.set(loc.id, loc.state);
+ if (loc.state === 'pending') report.counts.remote_unverified++;
+ continue;
+ }
+ let h;
+ try { h = await head(loc.provider, loc.key, loc.bucket); } catch { h = undefined; report.counts.remote_unverifiable++; effective.set(loc.id, loc.state); continue; }
+ if (h === undefined) { report.counts.remote_unverifiable++; effective.set(loc.id, loc.state); continue; }
+ report.counts.remote_checked++;
+ if (!h) {
+ effective.set(loc.id, 'missing');
+ if (live) issue('remote_missing', { object_id: obj.id, provider: loc.provider, key: loc.key, legacy_ref: obj.legacy_ref });
+ write(loc, 'missing', null);
+ continue;
+ }
+ let state = 'present';
+ if (sizeMatters && Number(h.size) !== expected) {
+ state = 'corrupt';
+ issue('size_mismatch', { object_id: obj.id, provider: loc.provider, expected, actual: Number(h.size), legacy_ref: obj.legacy_ref });
+ }
+ effective.set(loc.id, state);
+ write(loc, state, Number(h.size));
+ }
+
+ // Per object: is the canonical copy there, and is any copy there at all?
+ for (const obj of objects) {
+ if (obj.lifecycle_status !== 'ready') continue;
+ const locs = byObject.get(obj.id) || [];
+ const canon = obj.canonical_provider ? locs.find(l => l.provider === obj.canonical_provider) : null;
+ const present = locs.filter(l => effective.get(l.id) === 'present');
+ if (!canon) {
+ issue('no_canonical_location', { object_id: obj.id, kind: obj.kind, canonical_provider: obj.canonical_provider, legacy_ref: obj.legacy_ref });
+ } else if (['missing', 'corrupt'].includes(effective.get(canon.id))) {
+ if (present.length) issue('canonical_missing_replica_present', { object_id: obj.id, canonical: canon.provider, present: present.map(l => l.provider), legacy_ref: obj.legacy_ref });
+ }
+ const knownGone = locs.length && locs.every(l => ['missing', 'corrupt'].includes(effective.get(l.id)));
+ if (knownGone) issue('no_present_copy', { object_id: obj.id, kind: obj.kind, legacy_ref: obj.legacy_ref });
+ }
+
+ // Deleted objects that the inherited routes still serve.
+ for (const p of PROJECTIONS) {
+ const rows = db.all(`SELECT t.* FROM ${p.table} t JOIN media_objects o ON o.id = t.object_id WHERE o.lifecycle_status = 'deleted'${appId ? ' AND o.app_id = ?' : ''}`, appId ? [appId] : []);
+ for (const r of rows) if (p.open(r)) issue('deleted_publicly_reachable', { object_id: r.object_id, table: p.table, id: r.id ?? r.key, reason: 'the inherited row still serves it' });
+ }
+ // …and derivatives (thumbnails) of deleted objects that are still served by name.
+ const orphanThumbs = db.all(`SELECT d.* FROM media_variants v JOIN media_objects p ON p.id = v.object_id JOIN media_objects d ON d.id = v.derived_object_id
+ WHERE p.lifecycle_status = 'deleted' AND d.lifecycle_status != 'deleted' AND d.visibility != 'private'${appId ? ' AND p.app_id = ?' : ''}`, appId ? [appId] : []);
+ for (const d of orphanThumbs) {
+ const loc = (byObject.get(d.id) || []).find(l => effective.get(l.id) === 'present');
+ if (loc) issue('deleted_publicly_reachable', { object_id: d.id, kind: d.kind, legacy_ref: d.legacy_ref, reason: 'derivative of a deleted object is still served' });
+ }
+
+ // Inherited rows the model does not cover yet.
+ const gaps = [
+ ['vods', 'id', 'object_id IS NULL AND COALESCE(clips_only, 0) = 0'],
+ ['clips', 'id', 'object_id IS NULL'],
+ ['files', 'key', 'object_id IS NULL'],
+ ['pastes', 'id', "object_id IS NULL AND type = 'screenshot' AND screenshot_path IS NOT NULL"],
+ ];
+ for (const [table, keyCol, cond] of gaps) {
+ for (const r of db.all(`SELECT ${keyCol} AS k FROM ${table} WHERE ${cond}${appId ? ' AND app_id = ?' : ''}`, appId ? [appId] : [])) {
+ issue('missing_projection', { table, id: r.k });
+ }
+ }
+
+ report.total_issues = ISSUES.reduce((n, k) => n + report.issues[k].count, 0);
+ return report;
+}
+
+function summarize(report) {
+ const c = report.counts;
+ const lines = [`objects ${c.objects}, locations ${c.locations} (local checked ${c.local_checked}, remote checked ${c.remote_checked}, `
+ + `remote unverified ${c.remote_unverified}, unverifiable ${c.remote_unverifiable}${report.mode.verify ? `, updated ${c.locations_updated}` : ''})`];
+ for (const k of ISSUES) if (report.issues[k].count) lines.push(` ${k}: ${report.issues[k].count}`);
+ if (!report.total_issues) lines.push(' no issues');
+ return lines.join('\n');
+}
+
+module.exports = { reconcile, summarize, ISSUES, defaultHead };
new file mode 100644
new file mode 100644
@@ -0,0 +1,409 @@
@@ -0,0 +1,409 @@
+/**
+ * OpenVibe.Media — Object API v2 (mounted at /api/v2/:app/objects; docs/object-model.md)
+ *
+ * POST / init upload → { id, object, upload: { method, url, token, expires_at, max_bytes } }
+ * PUT /:id/content the bytes (single part; sha256 computed; size + quota checked)
+ * POST /:id/complete verify (expected hash, invariant) → lifecycle ready
+ * GET /:id metadata (id = med_… or legacy:<app>:<kind>:<id>)
+ * GET / cursor list (?kind&visibility&status&owner&user_id&include_deleted&limit&cursor)
+ * DELETE /:id soft delete (bytes kept MEDIA_DELETE_RETENTION_DAYS); 409 when held
+ * POST /:id/restore undo a soft delete inside the retention period
+ * GET /:id/download 302 to the public location, or a signed short-lived URL for private objects
+ * GET /:id/holds retention holds (?all=1 includes released)
+ * POST /:id/holds place a hold { kind, reason, created_by } (app key only)
+ * DELETE /:id/holds/:holdId release it (app key only)
+ *
+ * Auth: the app's API key, or a Network service token granting media.object.upload
+ * (writes) / media.object.read (reads) for namespace = :app. X-OV-Subject names the
+ * owner (usr_…); X-OV-User-Id (app key only) acts as one of the app's users. The
+ * content PUT and complete also accept the upload token handed out at init, so a
+ * browser can send the bytes directly.
+ *
+ * Public bytes: GET /o/:id (publicRouter) — public/unlisted objects openly,
+ * private ones only with a valid ?exp&sig from /download.
+ */
+'use strict';
+
+const express = require('express');
+const fs = require('fs');
+const path = require('path');
+const crypto = require('crypto');
+const { ids, http } = require('openvibe-contracts');
+const db = require('../db/database');
+const config = require('../config');
+const model = require('./model');
+const invariant = require('./invariant');
+const signing = require('./signing');
+const { tenantAuth, tenantCors } = require('../auth');
+const { sendWebhook } = require('../webhooks');
+
+const MB = 1024 * 1024;
+const upload = tenantAuth({ capability: 'media.object.upload' });
+const read = tenantAuth({ capability: 'media.object.read' });
+const appOnly = tenantAuth();
+
+function problem(res, status, code, detail, extra) {
+ return http.sendProblem(res, status, code, { detail, extra });
+}
+
+/** Owner subject from X-OV-Subject: 'usr_…' or 'user:usr_…'. undefined = malformed. */
+function subjectFrom(req) {
+ const raw = req.headers['x-ov-subject'];
+ if (raw == null || raw === '') return null;
+ const s = String(raw).trim();
+ const parsed = ids.parseSubject(s);
+ if (parsed && parsed.type === 'user') return parsed.id;
+ return ids.isSubjectId('user', s) ? s : undefined;
+}
+
+// An app acting for one of its users (X-OV-User-Id) sees public/unlisted objects
+// plus that user's own; the app itself and service principals see the namespace.
+function canSee(req, obj) {
+ return req.authType !== 'user' || obj.visibility !== 'private' || (obj.owner_user_id != null && obj.owner_user_id === req.userId);
+}
+function canWrite(req, obj) {
+ return req.authType !== 'user' || (obj.owner_user_id != null && obj.owner_user_id === req.userId);
+}
+
+function load(req, res) {
+ const obj = model.resolveObject(String(req.params.id || ''), req.appId);
+ if (!obj || !canSee(req, obj)) { problem(res, 404, 'media.object.not_found', 'No such object in this namespace'); return null; }
+ return obj;
+}
+
+/** Upload token (from init) or the usual tenant credential with media.object.upload. */
+function contentAuth(req, res, next) {
+ const token = req.query.token || req.headers['x-upload-token'];
+ if (!token) return upload(req, res, next);
+ const app = db.getApp(String(req.params.app || ''));
+ if (!app) return res.status(404).json({ error: 'Unknown app' });
+ if (!signing.verifyUploadToken(String(req.params.id || ''), token)) return problem(res, 401, 'media.upload_token.invalid', 'Upload token is invalid or expired');
+ req.appId = app.app_id;
+ req.appRow = app;
+ req.authType = 'upload_token';
+ next();
+}
+
+function sanitizeFilename(name) {
+ if (name == null || name === '') return null;
+ return path.basename(String(name)).replace(/[^\w.\- ]/g, '_').slice(0, 200) || null;
+}
+
+function quotaCheck(req, res, addBytes, excludeId = null) {
+ const quota = Number(req.appRow.quota_bytes) || 0;
+ if (!quota) return true;
+ const used = model.usedBytes(req.appId, excludeId);
+ if (used + addBytes <= quota) return true;
+ problem(res, 413, 'media.quota.exceeded', 'App storage quota exceeded', { quota_bytes: quota, used_bytes: used });
+ return false;
+}
+
+// ── Content upload (mounted ahead of the JSON body parser — see index.js) ──
+
+/** Stream the request body to `tmp`, hashing as it goes; stop past `cap` bytes. */
+function receive(req, tmp, cap) {
+ return new Promise((resolve) => {
+ const hash = crypto.createHash('sha256');
+ const out = fs.createWriteStream(tmp);
+ let n = 0, done = false;
+ const finish = (r) => { if (!done) { done = true; resolve(r); } };
+ // Failures settle once the file handle is closed, so the caller's unlink cannot race the open.
+ const fail = (r) => { if (done) return; if (out.closed) return finish(r); out.once('close', () => finish(r)); out.destroy(); };
+ req.on('data', (chunk) => {
+ if (done) return;
+ n += chunk.length;
+ if (n > cap) { req.unpipe(out); req.resume(); return fail({ error: 'too_large' }); }
+ hash.update(chunk);
+ });
+ req.on('aborted', () => fail({ error: 'aborted' }));
+ req.on('error', () => fail({ error: 'aborted' }));
+ out.on('error', (e) => fail({ error: 'write', message: e.message }));
+ out.on('finish', () => finish({ bytes: n, sha256: hash.digest('hex') }));
+ req.pipe(out);
+ });
+}
+
+async function putContent(req, res) {
+ const obj = load(req, res);
+ if (!obj) { req.resume(); return; }
+ if (!canWrite(req, obj)) { req.resume(); return problem(res, 403, 'media.object.forbidden', 'Not your object'); }
+ if (obj.legacy_ref) { req.resume(); return problem(res, 409, 'media.object.legacy_managed', 'This object is written through the v1 API'); }
+ if (obj.lifecycle_status !== 'uploading') { req.resume(); return problem(res, 409, 'media.object.not_uploading', `Object is ${obj.lifecycle_status}`); }
+
+ const declared = Number(obj.size_bytes) > 0 ? Number(obj.size_bytes) : null;
+ const cap = declared || config.objects.maxUploadMb * MB;
+ const len = Number(req.headers['content-length']);
+ if (Number.isFinite(len) && len > cap) {
+ res.set('Connection', 'close'); req.resume();
+ return problem(res, 413, 'media.object.too_large', `At most ${cap} bytes`);
+ }
+
+ const tmpDir = path.join(config.objects.path, '.tmp');
+ fs.mkdirSync(tmpDir, { recursive: true });
+ const tmp = path.join(tmpDir, `${obj.id}-${crypto.randomBytes(4).toString('hex')}`);
+ const got = await receive(req, tmp, cap);
+ const drop = () => { try { fs.unlinkSync(tmp); } catch { /* not written */ } };
+ if (got.error) {
+ drop();
+ if (got.error === 'too_large') { res.set('Connection', 'close'); return problem(res, 413, 'media.object.too_large', `At most ${cap} bytes`); }
+ return problem(res, 400, 'media.object.upload_interrupted', got.message || 'Upload did not complete');
+ }
+ if (declared && got.bytes !== declared) { drop(); return problem(res, 400, 'media.object.size_mismatch', `Declared ${declared} bytes, received ${got.bytes}`); }
+ if (!got.bytes) { drop(); return problem(res, 400, 'media.object.empty', 'No bytes received'); }
+ if (!quotaCheck(req, res, got.bytes, obj.id)) { drop(); return; }
+
+ try {
+ const dest = model.objectFilePath(obj);
+ fs.mkdirSync(path.dirname(dest), { recursive: true });
+ fs.renameSync(tmp, dest);
+ const contentType = String(req.headers['content-type'] || '').split(';')[0].trim().toLowerCase();
+ db.getDb().transaction(() => {
+ model.updateObject(obj.id, {
+ size_bytes: got.bytes, content_hash: got.sha256,
+ mime_type: obj.mime_type || (/^[\w.+-]+\/[\w.+-]+$/.test(contentType) ? contentType : 'application/octet-stream'),
+ canonical_provider: 'local', canonical_key: dest,
+ });
+ model.upsertLocation(obj.id, { provider: 'local', key: dest, state: 'present', size_bytes: got.bytes, checksum: got.sha256, verified: true });
+ })();
+ res.json({ id: obj.id, size_bytes: got.bytes, content_hash: got.sha256 });
+ } catch (err) {
+ drop();
+ console.error('[Objects] content store error:', err.message);
+ problem(res, 500, 'media.object.store_failed', 'Failed to store content');
+ }
+}
+
+// ── API router ───────────────────────────────────────────────
+
+const router = express.Router({ mergeParams: true });
+router.use(tenantCors);
+
+router.post('/', upload, (req, res) => {
+ try {
+ const b = req.body || {};
+ const kind = String(b.kind || 'file');
+ if (!model.KINDS.includes(kind)) return problem(res, 400, 'media.object.invalid', `kind must be one of ${model.KINDS.join(', ')}`);
+ const visibility = String(b.visibility || 'private');
+ if (!model.VISIBILITIES.includes(visibility)) return problem(res, 400, 'media.object.invalid', 'visibility must be public, unlisted or private');
+ let size = 0;
+ if (b.size_bytes != null) {
+ size = Number(b.size_bytes);
+ if (!Number.isInteger(size) || size < 1) return problem(res, 400, 'media.object.invalid', 'size_bytes must be a positive integer');
+ }
+ const max = config.objects.maxUploadMb * MB;
+ if (size > max) return problem(res, 413, 'media.object.too_large', `Single-part uploads are limited to ${max} bytes`);
+ if (size && invariant.wouldViolate({ kind, visibility, size_bytes: size })) {
+ return problem(res, 422, 'media.invariant.public_object_too_large', `Public playback objects are limited to ${config.objects.publicMaxMb} MB`);
+ }
+ if (b.content_hash != null && !/^[a-f0-9]{64}$/i.test(String(b.content_hash))) return problem(res, 400, 'media.object.invalid', 'content_hash must be a sha256 hex digest');
+ if (b.metadata != null && (typeof b.metadata !== 'object' || Array.isArray(b.metadata) || JSON.stringify(b.metadata).length > 16384)) {
+ return problem(res, 400, 'media.object.invalid', 'metadata must be an object of at most 16 KB');
+ }
+ const subject = subjectFrom(req);
+ if (subject === undefined) return problem(res, 400, 'media.object.invalid', 'X-OV-Subject must be a user subject (usr_…)');
+ if (!quotaCheck(req, res, size)) return;
+
+ const mime = b.mime_type && /^[\w.+-]+\/[\w.+-]+$/.test(String(b.mime_type)) ? String(b.mime_type).toLowerCase() : null;
+ const userId = req.authType === 'user' ? req.userId : (Number.isInteger(Number(b.user_id)) && Number(b.user_id) > 0 ? Number(b.user_id) : null);
+ const metadata = { ...(b.metadata || {}) };
+ const filename = sanitizeFilename(b.filename);
+ if (filename) metadata.filename = filename;
+ if (b.content_hash) metadata.expected_sha256 = String(b.content_hash).toLowerCase();
+ if (req.principal) metadata.created_by = req.principal.sub;
+
+ const id = model.createObject({
+ app_id: req.appId, namespace: req.appId, kind, owner_subject: subject, owner_app: req.appId, owner_user_id: userId,
+ visibility, lifecycle_status: 'uploading', mime_type: mime, size_bytes: size, metadata,
+ });
+ const tok = signing.uploadToken(id);
+ res.status(201).json({
+ id,
+ object: model.objectPublic(model.getObject(id)),
+ upload: {
+ method: 'PUT',
+ url: `${config.publicUrl}/api/v2/${encodeURIComponent(req.appId)}/objects/${id}/content?token=${encodeURIComponent(tok.token)}`,
+ token: tok.token,
+ expires_at: tok.expires_at,
+ max_bytes: size || max,
+ complete_url: `${config.publicUrl}/api/v2/${encodeURIComponent(req.appId)}/objects/${id}/complete`,
+ },
+ });
+ } catch (err) {
+ console.error('[Objects] init error:', err.message);
+ problem(res, 500, 'media.object.init_failed', 'Failed to create object');
+ }
+});
+
+router.post('/:id/complete', contentAuth, tenantCors, (req, res) => {
+ try {
+ const obj = load(req, res);
+ if (!obj) return;
+ if (!canWrite(req, obj)) return problem(res, 403, 'media.object.forbidden', 'Not your object');
+ if (obj.lifecycle_status === 'ready' && !obj.legacy_ref) return res.json(model.objectPublic(obj));
+ if (obj.legacy_ref || obj.lifecycle_status !== 'uploading') return problem(res, 409, 'media.object.not_uploading', `Object is ${obj.lifecycle_status}`);
+ const loc = model.listLocations(obj.id).find(l => l.provider === 'local' && l.state === 'present');
+ if (!loc || !fs.existsSync(loc.key)) return problem(res, 409, 'media.object.no_content', 'PUT the content first');
+ const md = model.parseJson(obj.metadata, {});
+ const expected = String((req.body && req.body.content_hash) || md.expected_sha256 || '').toLowerCase();
+ if (expected && expected !== obj.content_hash) {
+ return problem(res, 422, 'media.object.hash_mismatch', 'Content hash does not match', { expected, actual: obj.content_hash });
+ }
+ if (invariant.wouldViolate(obj)) {
+ return problem(res, 422, 'media.invariant.public_object_too_large', `Public playback objects are limited to ${config.objects.publicMaxMb} MB`);
+ }
+ if (!quotaCheck(req, res, Number(obj.size_bytes) || 0, obj.id)) return;
+ model.updateObject(obj.id, { lifecycle_status: 'ready' });
+ const done = model.getObject(obj.id);
+ invariant.record(done);
+ const body = model.objectPublic(done);
+ sendWebhook(req.appId, 'media.object.uploaded', body).catch(() => {});
+ res.json(body);
+ } catch (err) {
+ console.error('[Objects] complete error:', err.message);
+ problem(res, 500, 'media.object.complete_failed', 'Failed to complete upload');
+ }
+});
+
+router.get('/', read, (req, res) => {
+ try {
+ const q = req.query;
+ const limit = Math.min(Math.max(parseInt(q.limit || '50', 10) || 50, 1), 200);
+ const conds = ['app_id = ?'], params = [req.appId];
+ if (q.cursor) {
+ if (!/^med_[0-9A-HJKMNP-TV-Z]{26}$/.test(String(q.cursor))) return problem(res, 400, 'media.object.invalid', 'Bad cursor');
+ conds.push('id < ?'); params.push(String(q.cursor));
+ }
+ if (q.kind) { conds.push('kind = ?'); params.push(String(q.kind)); }
+ if (q.visibility) { conds.push('visibility = ?'); params.push(String(q.visibility)); }
+ if (q.status) { conds.push('lifecycle_status = ?'); params.push(String(q.status)); }
+ else if (!['1', 'true'].includes(String(q.include_deleted || ''))) conds.push("lifecycle_status != 'deleted'");
+ if (q.owner) { conds.push('owner_subject = ?'); params.push(String(q.owner).replace(/^user:/, '')); }
+ if (q.user_id != null) { conds.push('owner_user_id = ?'); params.push(Number(q.user_id)); }
+ if (req.authType === 'user') { conds.push("(visibility != 'private' OR owner_user_id = ?)"); params.push(req.userId); }
+ const rows = db.all(`SELECT * FROM media_objects WHERE ${conds.join(' AND ')} ORDER BY id DESC LIMIT ?`, [...params, limit + 1]);
+ const page = rows.slice(0, limit);
+ res.json({ objects: page.map(o => model.objectPublic(o, { locations: false })), next_cursor: rows.length > limit ? page[page.length - 1].id : null, limit });
+ } catch (err) {
+ console.error('[Objects] list error:', err.message);
+ problem(res, 500, 'media.object.list_failed', 'Failed to list objects');
+ }
+});
+
+router.get('/:id', read, (req, res) => {
+ const obj = load(req, res);
+ if (obj) res.json(model.objectPublic(obj));
+});
+
+router.delete('/:id', upload, (req, res) => {
+ try {
+ const obj = load(req, res);
+ if (!obj) return;
+ if (!canWrite(req, obj)) return problem(res, 403, 'media.object.forbidden', 'Not your object');
+ if (obj.legacy_ref) return problem(res, 409, 'media.object.legacy_managed', 'Delete this object through its v1 route (vods, clips, files, pastes)');
+ if (model.isHeld(obj.id)) return problem(res, 409, 'media.object.held', 'Object is under a retention hold');
+ if (obj.lifecycle_status === 'deleted') return res.json(model.objectPublic(obj));
+ res.json(model.objectPublic(model.softDelete(obj, { by: req.principal ? req.principal.sub : `app:${req.appId}` })));
+ } catch (err) {
+ if (err.code === 'media.object.held' || /retention hold/.test(err.message)) return problem(res, 409, 'media.object.held', 'Object is under a retention hold');
+ console.error('[Objects] delete error:', err.message);
+ problem(res, 500, 'media.object.delete_failed', 'Failed to delete object');
+ }
+});
+
+router.post('/:id/restore', upload, (req, res) => {
+ const obj = load(req, res);
+ if (!obj) return;
+ if (!canWrite(req, obj)) return problem(res, 403, 'media.object.forbidden', 'Not your object');
+ if (obj.lifecycle_status !== 'deleted' || obj.legacy_ref) return problem(res, 409, 'media.object.not_deleted', 'Only soft-deleted native objects can be restored');
+ const back = model.restore(obj);
+ if (!back) return problem(res, 410, 'media.object.purged', 'The retention period has passed and the bytes are gone');
+ res.json(model.objectPublic(back));
+});
+
+router.get('/:id/download', read, (req, res) => {
+ const obj = load(req, res);
+ if (!obj) return;
+ if (obj.lifecycle_status === 'deleted') return problem(res, 410, 'media.object.deleted', 'Object was deleted');
+ if (obj.lifecycle_status !== 'ready') return problem(res, 409, 'media.object.not_ready', `Object is ${obj.lifecycle_status}`);
+ const json = req.query.format === 'json';
+ res.set('Cache-Control', 'private, no-store');
+ if (obj.visibility !== 'private') {
+ const url = model.legacyPublicUrl(obj) || `${config.publicUrl}/o/${obj.id}`;
+ return json ? res.json({ url, expires_at: null, public: true }) : res.redirect(302, url);
+ }
+ const signed = signing.signedDownloadUrl(obj.id, req.query.ttl);
+ if (['1', 'true'].includes(String(req.query.redirect || ''))) return res.redirect(302, signed.url);
+ res.json({ url: signed.url, expires_at: signed.expires_at, public: false });
+});
+
+// ── Retention holds ──────────────────────────────────────────
+
+router.get('/:id/holds', read, (req, res) => {
+ const obj = load(req, res);
+ if (!obj) return;
+ res.json({ object_id: obj.id, holds: model.listHolds(obj.id, { includeReleased: ['1', 'true'].includes(String(req.query.all || '')) }) });
+});
+
+router.post('/:id/holds', appOnly, (req, res) => {
+ const obj = load(req, res);
+ if (!obj) return;
+ const b = req.body || {};
+ if (!model.HOLD_KINDS.includes(b.kind)) return problem(res, 400, 'media.hold.invalid', `kind must be one of ${model.HOLD_KINDS.join(', ')}`);
+ const by = b.created_by ? String(b.created_by).slice(0, 200) : `app:${req.appId}${req.userId != null ? `:user:${req.userId}` : ''}`;
+ res.status(201).json(model.placeHold({ object_id: obj.id, kind: b.kind, reason: b.reason || '', created_by: by }));
+});
+
+router.delete('/:id/holds/:holdId', appOnly, (req, res) => {
+ const obj = load(req, res);
+ if (!obj) return;
+ const hold = db.get('SELECT * FROM media_holds WHERE id = ? AND object_id = ?', [parseInt(req.params.holdId, 10), obj.id]);
+ if (!hold) return problem(res, 404, 'media.hold.not_found', 'No such hold on this object');
+ const by = (req.body && req.body.released_by) ? String(req.body.released_by).slice(0, 200) : `app:${req.appId}`;
+ res.json(model.releaseHold(hold.id, by));
+});
+
+// ── Public bytes: GET /o/:id ─────────────────────────────────
+
+const INLINE = /^(image\/(?!svg)|video\/|audio\/|application\/pdf$|text\/plain$)/;
+
+const publicRouter = express.Router();
+publicRouter.get('/:id', async (req, res) => {
+ try {
+ const obj = model.getObject(String(req.params.id || ''));
+ if (!obj) return res.status(404).json({ error: 'Not found' });
+ if (obj.lifecycle_status === 'deleted') return res.status(410).json({ error: 'Gone' });
+ if (obj.lifecycle_status !== 'ready') return res.status(404).json({ error: 'Not found' });
+ const signed = !!req.query.sig && signing.verifyDownload(obj.id, req.query.exp, req.query.sig);
+ // Private objects are indistinguishable from missing ones without a valid signature.
+ if (obj.visibility === 'private' && !signed) return res.status(404).json({ error: 'Not found' });
+
+ const mime = obj.mime_type || 'application/octet-stream';
+ const md = model.parseJson(obj.metadata, {});
+ const name = String(md.filename || obj.id).replace(/["\\\r\n]/g, '_');
+ const headers = {
+ 'Content-Type': mime,
+ 'X-Content-Type-Options': 'nosniff',
+ 'X-Robots-Tag': 'noindex',
+ 'Cache-Control': obj.visibility === 'private' ? 'private, no-store' : 'public, max-age=3600',
+ 'Content-Disposition': `${INLINE.test(mime) ? 'inline' : 'attachment'}; filename="${name}"`,
+ };
+ const locs = model.listLocations(obj.id);
+ const local = locs.find(l => l.provider === 'local' && l.state !== 'missing' && fs.existsSync(l.key));
+ if (local) return require('../public/routes').streamFileWithRange(req, res, local.key, headers);
+ const vodStorage = require('../vod/vod-storage');
+ for (const p of ['r2', 'b2']) {
+ const l = locs.find(x => x.provider === p && x.state !== 'missing' && x.state !== 'corrupt');
+ if (!l || !vodStorage.providerConfigured(p)) continue;
+ const url = await vodStorage.presignGet(p, l.key, 300).catch(() => null);
+ if (url) { res.set('Cache-Control', 'private, max-age=0'); res.set('X-Robots-Tag', 'noindex'); return res.redirect(302, url); }
+ }
+ res.status(404).json({ error: 'Object bytes unavailable' });
+ } catch (err) {
+ console.error('[Objects] /o error:', err.message);
+ if (!res.headersSent) res.status(500).json({ error: 'Failed to serve object' });
+ }
+});
+
+module.exports = router;
+module.exports.contentHandlers = [contentAuth, tenantCors, putContent];
+module.exports.publicRouter = publicRouter;
new file mode 100644
new file mode 100644
@@ -0,0 +1,65 @@
@@ -0,0 +1,65 @@
+/**
+ * OpenVibe.Media — signed object URLs and upload tokens
+ *
+ * HMAC-SHA256 over (purpose, object id, expiry) with MEDIA_SIGNING_SECRET.
+ * Purposes are separate so a download signature can never be replayed as an
+ * upload token or the other way round. Without the env secret a random
+ * per-process secret is used (and a warning logged): links then stop working
+ * at the next restart, which is safe but not what production wants.
+ */
+'use strict';
+
+const crypto = require('crypto');
+const config = require('../config');
+
+let _ephemeral = null;
+function secret() {
+ if (config.objects.signingSecret) return config.objects.signingSecret;
+ if (!_ephemeral) {
+ _ephemeral = crypto.randomBytes(32).toString('hex');
+ console.warn('[Objects] MEDIA_SIGNING_SECRET is not set — signed URLs and upload tokens die with this process');
+ }
+ return _ephemeral;
+}
+
+function mac(purpose, objectId, exp) {
+ return crypto.createHmac('sha256', secret()).update(`${purpose}\n${objectId}\n${exp}`).digest('base64url');
+}
+
+function safeEqual(a, b) {
+ const x = Buffer.from(String(a || '')), y = Buffer.from(String(b || ''));
+ return x.length === y.length && x.length > 0 && crypto.timingSafeEqual(x, y);
+}
+
+function nowS() { return Math.floor(Date.now() / 1000); }
+
+/** Short-lived GET URL for /o/:id. ttl is clamped to [30 s, 1 h]. */
+function signedDownloadUrl(objectId, ttlS = config.objects.signedUrlTtlS) {
+ const exp = nowS() + Math.min(3600, Math.max(30, Number(ttlS) || config.objects.signedUrlTtlS));
+ return {
+ url: `${config.publicUrl}/o/${encodeURIComponent(objectId)}?exp=${exp}&sig=${mac('get', objectId, exp)}`,
+ expires_at: new Date(exp * 1000).toISOString(),
+ };
+}
+
+function verifyDownload(objectId, exp, sig) {
+ const e = Number(exp);
+ if (!Number.isInteger(e) || e < nowS()) return false;
+ return safeEqual(sig, mac('get', objectId, e));
+}
+
+/** Token for PUT /api/v2/:app/objects/:id/content: "<exp>.<mac>". */
+function uploadToken(objectId, ttlS = config.objects.uploadTokenTtlS) {
+ const exp = nowS() + Math.max(60, Number(ttlS) || 3600);
+ return { token: `${exp}.${mac('put', objectId, exp)}`, expires_at: new Date(exp * 1000).toISOString() };
+}
+
+function verifyUploadToken(objectId, token) {
+ const m = /^(\d+)\.([A-Za-z0-9_-]+)$/.exec(String(token || ''));
+ if (!m) return false;
+ const exp = Number(m[1]);
+ if (exp < nowS()) return false;
+ return safeEqual(m[2], mac('put', objectId, exp));
+}
+
+module.exports = { signedDownloadUrl, verifyDownload, uploadToken, verifyUploadToken };
@@ -247,6 +247,9 @@ function _getPasteScoped(req, res) {
@@ -247,6 +247,9 @@ function _getPasteScoped(req, res) {
// Fully remove a paste's screenshot from local disk AND any legacy B2 object.
// Fully remove a paste's screenshot from local disk AND any legacy B2 object.
function removePasteScreenshot(paste) {
function removePasteScreenshot(paste) {
if (!paste) return;
if (!paste) return;
+ // A held screenshot keeps its bytes (the row delete that follows is refused by the hold trigger).
+ const objects = require('../objects/model');
+ if (objects.isHeldRow(paste.object_id !== undefined ? paste : db.get('SELECT object_id FROM pastes WHERE id = ?', [paste.id]))) return;
if (paste.screenshot_path) {
if (paste.screenshot_path) {
try { fs.unlinkSync(paste.screenshot_path); } catch { /* ignore */ }
try { fs.unlinkSync(paste.screenshot_path); } catch { /* ignore */ }
}
}
@@ -413,6 +416,7 @@ router.post('/:slug/censor', tenantAuth(), (req, res) => {
@@ -413,6 +416,7 @@ router.post('/:slug/censor', tenantAuth(), (req, res) => {
if (paste.type !== 'screenshot' || !paste.screenshot_path) {
if (paste.type !== 'screenshot' || !paste.screenshot_path) {
return res.status(400).json({ error: 'Not a screenshot paste' });
return res.status(400).json({ error: 'Not a screenshot paste' });
}
}
+ if (require('../objects/model').isHeldRow(paste)) return res.status(409).json({ error: 'Screenshot is under a retention hold', code: 'media.object.held' });
const censorUpload = multer({
const censorUpload = multer({
storage: screenshotStorage,
storage: screenshotStorage,
@@ -432,6 +436,7 @@ router.post('/:slug/censor', tenantAuth(), (req, res) => {
@@ -432,6 +436,7 @@ router.post('/:slug/censor', tenantAuth(), (req, res) => {
try { fs.unlinkSync(paste.screenshot_path); } catch { /* */ }
try { fs.unlinkSync(paste.screenshot_path); } catch { /* */ }
db.run('UPDATE pastes SET screenshot_path = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ?',
db.run('UPDATE pastes SET screenshot_path = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ?',
[req.file.path, paste.id]);
[req.file.path, paste.id]);
+ require('../objects/model').safeSync('paste', paste.id);
res.json({ paste: pastePublic(db.getPasteBySlug(paste.slug, req.appId)) });
res.json({ paste: pastePublic(db.getPasteBySlug(paste.slug, req.appId)) });
} catch (err2) {
} catch (err2) {
console.error('[Pastes] Censor error:', err2.message);
console.error('[Pastes] Censor error:', err2.message);
@@ -526,6 +531,7 @@ router.post('/', tenantAuth({ allowUser: true }), screenshotUpload.single('scree
@@ -526,6 +531,7 @@ router.post('/', tenantAuth({ allowUser: true }), screenshotUpload.single('scree
);
);
const paste = db.getPasteBySlug(slug, req.appId);
const paste = db.getPasteBySlug(slug, req.appId);
+ require('../objects/model').safeSync('paste', paste.id);
return res.status(201).json({ id: paste.id, slug, url: `/p/${slug}`, paste: pastePublic(paste) });
return res.status(201).json({ id: paste.id, slug, url: `/p/${slug}`, paste: pastePublic(paste) });
}
}
@@ -513,3 +513,4 @@ router.get('/p/:slug/screenshot', (req, res) => {
@@ -513,3 +513,4 @@ router.get('/p/:slug/screenshot', (req, res) => {
});
});
module.exports = router;
module.exports = router;
+module.exports.streamFileWithRange = streamFileWithRange;
@@ -68,6 +68,7 @@ router.post('/:kind/:id', tenantAuth({ allowUser: true }), upload.single('thumbn
@@ -68,6 +68,7 @@ router.post('/:kind/:id', tenantAuth({ allowUser: true }), upload.single('thumbn
const url = `/t/${filename}`;
const url = `/t/${filename}`;
const table = kind === 'vod' ? 'vods' : 'clips';
const table = kind === 'vod' ? 'vods' : 'clips';
db.run(`UPDATE ${table} SET thumbnail_url = ? WHERE id = ?`, [url, numId]);
db.run(`UPDATE ${table} SET thumbnail_url = ? WHERE id = ?`, [url, numId]);
+ require('../objects/model').safeSync(kind, numId);
return res.json({ url });
return res.json({ url });
}
}
@@ -196,6 +196,7 @@ async function generateVodThumbnail(vodId, filePath, opts = {}) {
@@ -196,6 +196,7 @@ async function generateVodThumbnail(vodId, filePath, opts = {}) {
// Clean up the previous thumbnail for this VOD
// Clean up the previous thumbnail for this VOD
_removeOldThumb(db.get('SELECT thumbnail_url FROM vods WHERE id = ?', [vodId])?.thumbnail_url, thumbUrl);
_removeOldThumb(db.get('SELECT thumbnail_url FROM vods WHERE id = ?', [vodId])?.thumbnail_url, thumbUrl);
db.run('UPDATE vods SET thumbnail_url = ? WHERE id = ?', [thumbUrl, vodId]);
db.run('UPDATE vods SET thumbnail_url = ? WHERE id = ?', [thumbUrl, vodId]);
+ require('../objects/model').safeSync('vod', vodId); // the thumbnail object follows the picture
}
}
return thumbUrl;
return thumbUrl;
}
}
@@ -206,6 +207,7 @@ async function generateClipThumbnail(clipId, filePath) {
@@ -206,6 +207,7 @@ async function generateClipThumbnail(clipId, filePath) {
if (thumbUrl) {
if (thumbUrl) {
_removeOldThumb(db.get('SELECT thumbnail_url FROM clips WHERE id = ?', [clipId])?.thumbnail_url, thumbUrl);
_removeOldThumb(db.get('SELECT thumbnail_url FROM clips WHERE id = ?', [clipId])?.thumbnail_url, thumbUrl);
db.run('UPDATE clips SET thumbnail_url = ? WHERE id = ?', [thumbUrl, clipId]);
db.run('UPDATE clips SET thumbnail_url = ? WHERE id = ?', [thumbUrl, clipId]);
+ require('../objects/model').safeSync('clip', clipId);
}
}
return thumbUrl;
return thumbUrl;
}
}
@@ -77,6 +77,7 @@ async function _recutClip(clipId, { reason = 'recut' } = {}) {
@@ -77,6 +77,7 @@ async function _recutClip(clipId, { reason = 'recut' } = {}) {
if (!cut.ok) return fail(clipId, clip, attempt, cut.error);
if (!cut.ok) return fail(clipId, clip, attempt, cut.error);
db.run("UPDATE clips SET file_path = ?, duration_seconds = ?, end_time = ?, status = 'ready', cut_error = NULL, cut_next_at = NULL WHERE id = ?",
db.run("UPDATE clips SET file_path = ?, duration_seconds = ?, end_time = ?, status = 'ready', cut_error = NULL, cut_next_at = NULL WHERE id = ?",
[cut.filePath, cut.duration, startTime + cut.duration, clipId]);
[cut.filePath, cut.duration, startTime + cut.duration, clipId]);
+ require('../objects/model').safeSync('clip', clipId);
try { await require('../thumbnails/thumbnail-service').generateClipThumbnail(clipId, cut.filePath); } catch { /* */ }
try { await require('../thumbnails/thumbnail-service').generateClipThumbnail(clipId, cut.filePath); } catch { /* */ }
console.log(`[Clips] Clip ${clipId} ${reason} OK from vod ${clip.vod_id} (${startTime.toFixed(1)}-${(startTime + cut.duration).toFixed(1)}s, attempt ${attempt})`);
console.log(`[Clips] Clip ${clipId} ${reason} OK from vod ${clip.vod_id} (${startTime.toFixed(1)}-${(startTime + cut.duration).toFixed(1)}s, attempt ${attempt})`);
sendWebhook(clip.app_id, 'clip.ready', _clipPublic(db.getClipById(clipId))).catch(() => {});
sendWebhook(clip.app_id, 'clip.ready', _clipPublic(db.getClipById(clipId))).catch(() => {});
@@ -92,6 +93,7 @@ function fail(clipId, clip, attempt, error) {
@@ -92,6 +93,7 @@ function fail(clipId, clip, attempt, error) {
const mins = BACKOFF_MIN[Math.min(attempt - 1, BACKOFF_MIN.length - 1)];
const mins = BACKOFF_MIN[Math.min(attempt - 1, BACKOFF_MIN.length - 1)];
const nextAt = more ? new Date(Date.now() + mins * 60000).toISOString().replace('T', ' ').slice(0, 19) : null;
const nextAt = more ? new Date(Date.now() + mins * 60000).toISOString().replace('T', ' ').slice(0, 19) : null;
db.run("UPDATE clips SET status = 'failed', cut_error = ?, cut_next_at = ? WHERE id = ?", [msg, nextAt, clipId]);
db.run("UPDATE clips SET status = 'failed', cut_error = ?, cut_next_at = ? WHERE id = ?", [msg, nextAt, clipId]);
+ require('../objects/model').safeSync('clip', clipId);
console.warn(`[Clips] Clip ${clipId} attempt ${attempt}/${MAX_ATTEMPTS} failed: ${msg}${more ? ` — retry in ${mins} min` : ' — giving up'}`);
console.warn(`[Clips] Clip ${clipId} attempt ${attempt}/${MAX_ATTEMPTS} failed: ${msg}${more ? ` — retry in ${mins} min` : ' — giving up'}`);
sendWebhook(clip.app_id, 'clip.failed', _clipPublic(db.getClipById(clipId))).catch(() => {});
sendWebhook(clip.app_id, 'clip.failed', _clipPublic(db.getClipById(clipId))).catch(() => {});
return { ok: false, error: msg, retry_at: nextAt };
return { ok: false, error: msg, retry_at: nextAt };
@@ -23,6 +23,7 @@ const tools = require('./media-tools');
@@ -23,6 +23,7 @@ const tools = require('./media-tools');
const cutter = require('./clip-cutter');
const cutter = require('./clip-cutter');
const { tenantAuth, tenantCors } = require('../auth');
const { tenantAuth, tenantCors } = require('../auth');
const { sendWebhook } = require('../webhooks');
const { sendWebhook } = require('../webhooks');
+const objects = require('../objects/model');
const router = express.Router({ mergeParams: true });
const router = express.Router({ mergeParams: true });
router.use(tenantCors);
router.use(tenantCors);
@@ -197,6 +198,7 @@ router.post('/', tenantAuth({ allowUser: true }), clipUpload.single('video'), as
@@ -197,6 +198,7 @@ router.post('/', tenantAuth({ allowUser: true }), clipUpload.single('video'), as
if (cut.ok) {
if (cut.ok) {
db.run('UPDATE clips SET file_path = ?, duration_seconds = ?, end_time = ?, status = ? WHERE id = ?',
db.run('UPDATE clips SET file_path = ?, duration_seconds = ?, end_time = ?, status = ? WHERE id = ?',
[cut.filePath, cut.duration, startTime + cut.duration, 'ready', clipId]);
[cut.filePath, cut.duration, startTime + cut.duration, 'ready', clipId]);
+ objects.safeSync('clip', clipId);
try {
try {
await require('../thumbnails/thumbnail-service').generateClipThumbnail(clipId, cut.filePath);
await require('../thumbnails/thumbnail-service').generateClipThumbnail(clipId, cut.filePath);
} catch { /* */ }
} catch { /* */ }
@@ -207,6 +209,7 @@ router.post('/', tenantAuth({ allowUser: true }), clipUpload.single('video'), as
@@ -207,6 +209,7 @@ router.post('/', tenantAuth({ allowUser: true }), clipUpload.single('video'), as
// (attempt 2 pulls a cloud-stored VOD back to local disk first).
// (attempt 2 pulls a cloud-stored VOD back to local disk first).
const nextAt = new Date(Date.now() + 2 * 60000).toISOString().replace('T', ' ').slice(0, 19);
const nextAt = new Date(Date.now() + 2 * 60000).toISOString().replace('T', ' ').slice(0, 19);
db.run("UPDATE clips SET status = 'failed', cut_error = ?, cut_attempts = 1, cut_next_at = ? WHERE id = ?", [String(cut.error || 'cut failed').slice(0, 500), nextAt, clipId]);
db.run("UPDATE clips SET status = 'failed', cut_error = ?, cut_attempts = 1, cut_next_at = ? WHERE id = ?", [String(cut.error || 'cut failed').slice(0, 500), nextAt, clipId]);
+ objects.safeSync('clip', clipId);
console.warn(`[Clips] Clip ${clipId} failed: ${cut.error} — auto-retry in 2 min`);
console.warn(`[Clips] Clip ${clipId} failed: ${cut.error} — auto-retry in 2 min`);
sendWebhook(appId, 'clip.failed', clipPublic(db.getClipById(clipId))).catch(() => {});
sendWebhook(appId, 'clip.failed', clipPublic(db.getClipById(clipId))).catch(() => {});
}
}
@@ -351,6 +354,7 @@ router.put('/:id', tenantAuth(), (req, res) => {
@@ -351,6 +354,7 @@ router.put('/:id', tenantAuth(), (req, res) => {
const t = sanitizeClipTitle(title, '');
const t = sanitizeClipTitle(title, '');
if (!t) return res.status(400).json({ error: 'Title must be 1-200 characters' });
if (!t) return res.status(400).json({ error: 'Title must be 1-200 characters' });
db.run('UPDATE clips SET title = ? WHERE id = ?', [t, clip.id]);
db.run('UPDATE clips SET title = ? WHERE id = ?', [t, clip.id]);
+ if (visibility === undefined) objects.safeSync('clip', clip.id);
}
}
if (visibility !== undefined) db.setClipVisibility(clip.id, visibility);
if (visibility !== undefined) db.setClipVisibility(clip.id, visibility);
res.json({ clip: clipPublic(db.getClipById(clip.id, req.appId)) });
res.json({ clip: clipPublic(db.getClipById(clip.id, req.appId)) });
@@ -364,6 +368,7 @@ router.delete('/:id', tenantAuth(), (req, res) => {
@@ -364,6 +368,7 @@ router.delete('/:id', tenantAuth(), (req, res) => {
try {
try {
const clip = _getClipScoped(req, res);
const clip = _getClipScoped(req, res);
if (!clip) return;
if (!clip) return;
+ if (objects.isHeldRow(clip)) return res.status(409).json({ error: 'Clip is under a retention hold', code: 'media.object.held' });
// Delete the local file + any offloaded B2/R2 object (clips carry
// Delete the local file + any offloaded B2/R2 object (clips carry
// storage_provider/storage_key like VODs).
// storage_provider/storage_key like VODs).
@@ -136,6 +136,8 @@ async function finalizeVod(vodId, opts = {}) {
@@ -136,6 +136,8 @@ async function finalizeVod(vodId, opts = {}) {
return await _doFinalize(vodId, opts);
return await _doFinalize(vodId, opts);
} finally {
} finally {
_finalizing.delete(vodId);
_finalizing.delete(vodId);
+ // Every outcome (ready, quarantined, failed) re-projects the object; a deleted row was already marked by trigger.
+ require('../objects/model').safeSync('vod', vodId);
}
}
}
}
@@ -123,6 +123,10 @@ async function _scanPass(deep) {
@@ -123,6 +123,10 @@ async function _scanPass(deep) {
// Delete a VOD's files everywhere + its DB row. Mirrors the manual delete route.
// Delete a VOD's files everywhere + its DB row. Mirrors the manual delete route.
function _hardDeleteVod(vod) {
function _hardDeleteVod(vod) {
try {
try {
+ if (require('../objects/model').isHeldRow(db.get('SELECT object_id FROM vods WHERE id = ?', [vod.id]))) {
+ console.log(`[VOD-Health] vod ${vod.id} is under a retention hold — not deleting`);
+ return false;
+ }
if (vod.file_path) {
if (vod.file_path) {
try { require('./vod-storage').deleteVodObjects(vod).catch(() => {}); } catch { /* */ }
try { require('./vod-storage').deleteVodObjects(vod).catch(() => {}); } catch { /* */ }
// Local file + sidecars (.seekable.webm / .seekable.mp4, .master.mkv).
// Local file + sidecars (.seekable.webm / .seekable.mp4, .master.mkv).
@@ -218,6 +218,7 @@ class StreamRecorder {
@@ -218,6 +218,7 @@ class StreamRecorder {
this.activeRecordings.set(vod.id, recording);
this.activeRecordings.set(vod.id, recording);
db.run('UPDATE vods SET is_recording = 1, file_path = ? WHERE id = ?', [filePath, vod.id]);
db.run('UPDATE vods SET is_recording = 1, file_path = ? WHERE id = ?', [filePath, vod.id]);
if (masterPath) db.run('UPDATE vods SET master_file_path = ? WHERE id = ?', [masterPath, vod.id]);
if (masterPath) db.run('UPDATE vods SET master_file_path = ? WHERE id = ?', [masterPath, vod.id]);
+ require('../objects/model').safeSync('vod', vod.id);
return recording;
return recording;
}
}
@@ -25,6 +25,7 @@ const recorder = require('./recorder');
@@ -25,6 +25,7 @@ const recorder = require('./recorder');
const tools = require('./media-tools');
const tools = require('./media-tools');
const { finalizeVod, vodPublic } = require('./finalize');
const { finalizeVod, vodPublic } = require('./finalize');
const { tenantAuth, tenantCors } = require('../auth');
const { tenantAuth, tenantCors } = require('../auth');
+const objects = require('../objects/model');
const router = express.Router({ mergeParams: true });
const router = express.Router({ mergeParams: true });
router.use(tenantCors);
router.use(tenantCors);
@@ -101,6 +102,7 @@ router.post('/', tenantAuth(), (req, res) => {
@@ -101,6 +102,7 @@ router.post('/', tenantAuth(), (req, res) => {
const id = result.lastInsertRowid;
const id = result.lastInsertRowid;
if (clips_only) db.run('UPDATE vods SET clips_only = 1 WHERE id = ?', [id]);
if (clips_only) db.run('UPDATE vods SET clips_only = 1 WHERE id = ?', [id]);
if (visibility) db.setVodVisibility(id, visibility);
if (visibility) db.setVodVisibility(id, visibility);
+ else if (!clips_only) objects.safeSync('vod', id);
res.status(201).json({ id });
res.status(201).json({ id });
} catch (err) {
} catch (err) {
console.error('[VOD] Create error:', err.message);
console.error('[VOD] Create error:', err.message);
@@ -342,6 +344,7 @@ router.put('/:id', tenantAuth(), (req, res) => {
@@ -342,6 +344,7 @@ router.put('/:id', tenantAuth(), (req, res) => {
db.run(`UPDATE vods SET ${updates.join(', ')} WHERE id = ?`, params);
db.run(`UPDATE vods SET ${updates.join(', ')} WHERE id = ?`, params);
}
}
if (visibility !== undefined) db.setVodVisibility(vod.id, visibility);
if (visibility !== undefined) db.setVodVisibility(vod.id, visibility);
+ else if (updates.length) objects.safeSync('vod', vod.id);
res.json({ vod: vodPublic(db.getVodById(vod.id, req.appId)) });
res.json({ vod: vodPublic(db.getVodById(vod.id, req.appId)) });
} catch (err) {
} catch (err) {
console.error('[VOD] Update error:', err.message);
console.error('[VOD] Update error:', err.message);
@@ -354,6 +357,7 @@ router.delete('/:id', tenantAuth(), (req, res) => {
@@ -354,6 +357,7 @@ router.delete('/:id', tenantAuth(), (req, res) => {
try {
try {
const vod = _getVodScoped(req, res);
const vod = _getVodScoped(req, res);
if (!vod) return;
if (!vod) return;
+ if (objects.isHeldRow(vod)) return res.status(409).json({ error: 'VOD is under a retention hold', code: 'media.object.held' });
if (recorder.isRecording(vod.id)) recorder.stopRecording(vod.id);
if (recorder.isRecording(vod.id)) recorder.stopRecording(vod.id);
activeChunkUploads.delete(vod.id);
activeChunkUploads.delete(vod.id);
@@ -160,6 +160,16 @@ function clientFor(name) {
@@ -160,6 +160,16 @@ function clientFor(name) {
return clients[name];
return clients[name];
}
}
+function bucketFor(name) {
+ return (PROVIDER_ENV[name] && PROVIDER_ENV[name].bucket) || null;
+}
+
+// Object model (Wave 4): holds freeze an object's placement, and every tier move
+// re-projects the VOD's media_locations. Lazy — the model requires this module.
+const objects = () => require('../objects/model');
+function _held(vod) { return objects().isHeldRow(vod); }
+function _moved(vodId, verified) { objects().afterTierMove(vodId, verified); }
+
function keyForVod(vod) {
function keyForVod(vod) {
if (vod.storage_key) return vod.storage_key;
if (vod.storage_key) return vod.storage_key;
return KEY_PREFIX + path.basename(vod.file_path || '');
return KEY_PREFIX + path.basename(vod.file_path || '');
@@ -374,6 +384,7 @@ async function moveToCold(vodId) {
@@ -374,6 +384,7 @@ async function moveToCold(vodId) {
const vod = db.get('SELECT * FROM vods WHERE id = ?', [vodId]);
const vod = db.get('SELECT * FROM vods WHERE id = ?', [vodId]);
if (!vod || !vod.file_path) return { ok: false, error: 'VOD not found' };
if (!vod || !vod.file_path) return { ok: false, error: 'VOD not found' };
if (vod.is_recording) return { ok: false, error: 'VOD is currently recording' };
if (vod.is_recording) return { ok: false, error: 'VOD is currently recording' };
+ if (_held(vod)) return { ok: false, held: true, error: 'VOD is under a retention hold' };
if (!providerConfigured('b2')) return { ok: false, error: 'B2 not configured' };
if (!providerConfigured('b2')) return { ok: false, error: 'B2 not configured' };
const key = keyForVod(vod);
const key = keyForVod(vod);
@@ -384,6 +395,7 @@ async function moveToCold(vodId) {
@@ -384,6 +395,7 @@ async function moveToCold(vodId) {
// Not local — maybe already offloaded
// Not local — maybe already offloaded
if (await headObject('b2', key)) {
if (await headObject('b2', key)) {
db.run("UPDATE vods SET storage_provider = 'b2', storage_key = ? WHERE id = ?", [key, vodId]);
db.run("UPDATE vods SET storage_provider = 'b2', storage_key = ? WHERE id = ?", [key, vodId]);
+ _moved(vodId, ['b2']);
return { ok: true, already: true };
return { ok: true, already: true };
}
}
// Neither here nor in B2: nothing to offload, ever. Quarantine the row so the
// Neither here nor in B2: nothing to offload, ever. Quarantine the row so the
@@ -403,6 +415,7 @@ async function moveToCold(vodId) {
@@ -403,6 +415,7 @@ async function moveToCold(vodId) {
console.error(`[VodStorage] Uploaded but failed to remove local file for VOD ${vodId}:`, err.message);
console.error(`[VodStorage] Uploaded but failed to remove local file for VOD ${vodId}:`, err.message);
}
}
cleanupSidecar(local);
cleanupSidecar(local);
+ _moved(vodId, ['b2']);
console.log(`[VodStorage] VOD ${vodId} offloaded to B2: ${key} (${(localSize / 1048576).toFixed(1)} MB)`);
console.log(`[VodStorage] VOD ${vodId} offloaded to B2: ${key} (${(localSize / 1048576).toFixed(1)} MB)`);
return { ok: true, bytes: localSize };
return { ok: true, bytes: localSize };
@@ -434,6 +447,7 @@ async function moveToHot(vodId) {
@@ -434,6 +447,7 @@ async function moveToHot(vodId) {
if (fs.existsSync(local)) {
if (fs.existsSync(local)) {
db.run("UPDATE vods SET storage_provider = 'local' WHERE id = ?", [vodId]);
db.run("UPDATE vods SET storage_provider = 'local' WHERE id = ?", [vodId]);
+ _moved(vodId, []);
return { ok: true, already: true };
return { ok: true, already: true };
}
}
@@ -483,6 +497,7 @@ async function moveToHot(vodId) {
@@ -483,6 +497,7 @@ async function moveToHot(vodId) {
// Restoring to local keeps the B2 canonical copy; drop any R2 copy.
// Restoring to local keeps the B2 canonical copy; drop any R2 copy.
if (providerOf(vod) === 'r2' && providerConfigured('r2')) await deleteObject('r2', key);
if (providerOf(vod) === 'r2' && providerConfigured('r2')) await deleteObject('r2', key);
db.run("UPDATE vods SET storage_provider = 'local' WHERE id = ?", [vodId]);
db.run("UPDATE vods SET storage_provider = 'local' WHERE id = ?", [vodId]);
+ _moved(vodId, []);
console.log(`[VodStorage] VOD ${vodId} restored to local (${(head.size / 1048576).toFixed(1)} MB)`);
console.log(`[VodStorage] VOD ${vodId} restored to local (${(head.size / 1048576).toFixed(1)} MB)`);
return { ok: true, bytes: head.size };
return { ok: true, bytes: head.size };
} catch (err) {
} catch (err) {
@@ -496,6 +511,7 @@ async function promoteToR2(vodId) {
@@ -496,6 +511,7 @@ async function promoteToR2(vodId) {
const vod = db.get('SELECT * FROM vods WHERE id = ?', [vodId]);
const vod = db.get('SELECT * FROM vods WHERE id = ?', [vodId]);
if (!vod || !vod.file_path) return { ok: false, error: 'VOD not found' };
if (!vod || !vod.file_path) return { ok: false, error: 'VOD not found' };
if (vod.is_recording) return { ok: false, error: 'VOD is currently recording' };
if (vod.is_recording) return { ok: false, error: 'VOD is currently recording' };
+ if (_held(vod)) return { ok: false, held: true, error: 'VOD is under a retention hold' };
if (!providerConfigured('r2') || providerHealthy.r2 === false) return { ok: false, error: 'R2 not available' };
if (!providerConfigured('r2') || providerHealthy.r2 === false) return { ok: false, error: 'R2 not available' };
if (!providerConfigured('b2')) return { ok: false, error: 'B2 not configured' };
if (!providerConfigured('b2')) return { ok: false, error: 'B2 not configured' };
@@ -519,6 +535,7 @@ async function promoteToR2(vodId) {
@@ -519,6 +535,7 @@ async function promoteToR2(vodId) {
}
}
db.run("UPDATE vods SET storage_provider = 'r2', storage_key = ? WHERE id = ?", [key, vodId]);
db.run("UPDATE vods SET storage_provider = 'r2', storage_key = ? WHERE id = ?", [key, vodId]);
+ _moved(vodId, ['b2', 'r2']);
// Popular VODs live in R2+B2; free the local copy
// Popular VODs live in R2+B2; free the local copy
let freed = 0;
let freed = 0;
@@ -540,6 +557,7 @@ async function demoteFromR2(vodId) {
@@ -540,6 +557,7 @@ async function demoteFromR2(vodId) {
const vod = db.get('SELECT * FROM vods WHERE id = ?', [vodId]);
const vod = db.get('SELECT * FROM vods WHERE id = ?', [vodId]);
if (!vod) return { ok: false, error: 'VOD not found' };
if (!vod) return { ok: false, error: 'VOD not found' };
if (providerOf(vod) !== 'r2') return { ok: true, already: true };
if (providerOf(vod) !== 'r2') return { ok: true, already: true };
+ if (_held(vod)) return { ok: false, held: true, error: 'VOD is under a retention hold' };
const key = keyForVod(vod);
const key = keyForVod(vod);
try {
try {
@@ -551,6 +569,7 @@ async function demoteFromR2(vodId) {
@@ -551,6 +569,7 @@ async function demoteFromR2(vodId) {
}
}
await deleteObject('r2', key);
await deleteObject('r2', key);
db.run("UPDATE vods SET storage_provider = 'b2' WHERE id = ?", [vodId]);
db.run("UPDATE vods SET storage_provider = 'b2' WHERE id = ?", [vodId]);
+ _moved(vodId, ['b2']);
console.log(`[VodStorage] VOD ${vodId} demoted from R2 to B2`);
console.log(`[VodStorage] VOD ${vodId} demoted from R2 to B2`);
return { ok: true };
return { ok: true };
} catch (err) {
} catch (err) {
@@ -561,6 +580,8 @@ async function demoteFromR2(vodId) {
@@ -561,6 +580,8 @@ async function demoteFromR2(vodId) {
/** Delete a VOD's (or clip's — same columns) media everywhere (local + B2 + R2). */
/** Delete a VOD's (or clip's — same columns) media everywhere (local + B2 + R2). */
async function deleteVodObjects(vod) {
async function deleteVodObjects(vod) {
if (!vod?.file_path) return;
if (!vod?.file_path) return;
+ // Last line of defence for every delete path: a held object keeps its bytes.
+ if (_held(vod)) { console.warn(`[VodStorage] Not deleting media of ${vod.object_id}: under a retention hold`); return; }
const local = localPathForVod(vod);
const local = localPathForVod(vod);
if (fs.existsSync(local)) {
if (fs.existsSync(local)) {
try { fs.unlinkSync(local); } catch { /* ignore */ }
try { fs.unlinkSync(local); } catch { /* ignore */ }
@@ -783,7 +804,7 @@ async function runSweep() {
@@ -783,7 +804,7 @@ async function runSweep() {
`, [`-${settings.minAgeDays} days`, settings.maxViewsForCold, `-${settings.minLastAccessDays} days`, settings.maxPerSweep]);
`, [`-${settings.minAgeDays} days`, settings.maxViewsForCold, `-${settings.minLastAccessDays} days`, settings.maxPerSweep]);
}
}
- let skippedBackoff = 0, quarantined = ghosts;
+ let skippedBackoff = 0, skippedHeld = 0, quarantined = ghosts;
for (const vod of candidates) {
for (const vod of candidates) {
if (underPressure && drainSatisfied(diskUsage(config.vod.path), settings)) break;
if (underPressure && drainSatisfied(diskUsage(config.vod.path), settings)) break;
const failedAt = offloadFailedAt.get(vod.id);
const failedAt = offloadFailedAt.get(vod.id);
@@ -794,6 +815,7 @@ async function runSweep() {
@@ -794,6 +815,7 @@ async function runSweep() {
const key = keyForVod(vod);
const key = keyForVod(vod);
if (await headObject('b2', key).catch(() => null)) {
if (await headObject('b2', key).catch(() => null)) {
db.run("UPDATE vods SET storage_provider = 'b2', storage_key = ? WHERE id = ?", [key, vod.id]);
db.run("UPDATE vods SET storage_provider = 'b2', storage_key = ? WHERE id = ?", [key, vod.id]);
+ _moved(vod.id, ['b2']);
} else {
} else {
quarantineMissing(vod.id, `file not found at ${localPathForVod(vod)}`);
quarantineMissing(vod.id, `file not found at ${localPathForVod(vod)}`);
quarantined++;
quarantined++;
@@ -804,6 +826,8 @@ async function runSweep() {
@@ -804,6 +826,8 @@ async function runSweep() {
if (result.ok) {
if (result.ok) {
offloadFailedAt.delete(vod.id);
offloadFailedAt.delete(vod.id);
if (!result.already) { migrated++; bytesFreed += result.bytes || 0; }
if (!result.already) { migrated++; bytesFreed += result.bytes || 0; }
+ } else if (result.held) {
+ skippedHeld++;
} else {
} else {
offloadFailedAt.set(vod.id, Date.now());
offloadFailedAt.set(vod.id, Date.now());
errors.push({ id: vod.id, error: result.error });
errors.push({ id: vod.id, error: result.error });
@@ -812,10 +836,11 @@ async function runSweep() {
@@ -812,10 +836,11 @@ async function runSweep() {
// Say what happened whenever it matters: the old sweep only logged on success, so
// Say what happened whenever it matters: the old sweep only logged on success, so
// weeks of "draining" that freed nothing looked healthy in the journal.
// weeks of "draining" that freed nothing looked healthy in the journal.
- if (underPressure || errors.length || quarantined) {
+ if (underPressure || errors.length || quarantined || skippedHeld) {
const freedMb = (bytesFreed / 1048576).toFixed(1);
const freedMb = (bytesFreed / 1048576).toFixed(1);
console.log(`[VodStorage] Offload pass: ${candidates.length} candidate(s), ${migrated} uploaded (${freedMb} MB freed), `
console.log(`[VodStorage] Offload pass: ${candidates.length} candidate(s), ${migrated} uploaded (${freedMb} MB freed), `
+ `${errors.length} failed, ${skippedBackoff} in retry back-off, ${quarantined} quarantined`
+ `${errors.length} failed, ${skippedBackoff} in retry back-off, ${quarantined} quarantined`
+ + (skippedHeld ? `, ${skippedHeld} under a retention hold` : '')
+ (underPressure ? `, disk now ${diskUsage(config.vod.path).usePct}%` : ''));
+ (underPressure ? `, disk now ${diskUsage(config.vod.path).usePct}%` : ''));
for (const e of errors.slice(0, 5)) console.warn(`[VodStorage] VOD ${e.id}: ${e.error}`);
for (const e of errors.slice(0, 5)) console.warn(`[VodStorage] VOD ${e.id}: ${e.error}`);
if (underPressure && !migrated && !candidates.length) {
if (underPressure && !migrated && !candidates.length) {
@@ -877,6 +902,7 @@ async function runSweep() {
@@ -877,6 +902,7 @@ async function runSweep() {
for (const vod of popular) {
for (const vod of popular) {
const result = await promoteToR2(vod.id);
const result = await promoteToR2(vod.id);
if (result.ok && !result.already) { promoted++; bytesFreed += result.bytes || 0; }
if (result.ok && !result.already) { promoted++; bytesFreed += result.bytes || 0; }
+ else if (result.held) skippedHeld++;
else if (!result.ok) errors.push({ id: vod.id, error: result.error });
else if (!result.ok) errors.push({ id: vod.id, error: result.error });
}
}
@@ -903,6 +929,7 @@ async function runSweep() {
@@ -903,6 +929,7 @@ async function runSweep() {
critical,
critical,
stillNeedsDrain,
stillNeedsDrain,
skippedBackoff,
skippedBackoff,
+ skippedHeld,
quarantined,
quarantined,
diskPct: diskAfter.usePct,
diskPct: diskAfter.usePct,
freeGb: Number((diskAfter.available / GB).toFixed(1)),
freeGb: Number((diskAfter.available / GB).toFixed(1)),
@@ -978,6 +1005,7 @@ async function migrateLegacy() {
@@ -978,6 +1005,7 @@ async function migrateLegacy() {
const head = await headObject('b2', key);
const head = await headObject('b2', key);
if (head) {
if (head) {
db.run("UPDATE vods SET storage_provider = 'b2', storage_key = ? WHERE id = ?", [key, vod.id]);
db.run("UPDATE vods SET storage_provider = 'b2', storage_key = ? WHERE id = ?", [key, vod.id]);
+ _moved(vod.id, ['b2']);
flipped++;
flipped++;
} else if (fs.existsSync(localPathForVod(vod))) {
} else if (fs.existsSync(localPathForVod(vod))) {
restoredLocal++; // still local, sweep will re-offload
restoredLocal++; // still local, sweep will re-offload
@@ -1212,6 +1240,9 @@ module.exports = {
@@ -1212,6 +1240,9 @@ module.exports = {
uploadTimeoutMs,
uploadTimeoutMs,
providerOf,
providerOf,
isRemote,
isRemote,
+ providerConfigured,
+ bucketFor,
+ headObject,
keyForVod,
keyForVod,
localPathForVod,
localPathForVod,
resolvePlayback,
resolvePlayback,
new file mode 100644
new file mode 100644
@@ -0,0 +1,270 @@
@@ -0,0 +1,270 @@
+'use strict';
+// Object API v2 (/api/v2/:app/objects) and /o/:id: init → PUT → complete with an app key, an upload
+// token or a Network service token; sha256 + size + quota checks; cursor listing; signed private
+// downloads; soft delete/restore; retention holds; the public-size invariant at upload time; and the
+// legacy projections (read-only through v2; their v1 deletes honour holds).
+const assert = require('assert');
+const fs = require('fs');
+const os = require('os');
+const path = require('path');
+const http = require('http');
+const crypto = require('crypto');
+
+const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-objapi-'));
+const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+process.env.DB_PATH = path.join(tmp, 'media.db');
+process.env.VOD_PATH = dir('vods');
+process.env.FILES_PATH = dir('files');
+process.env.OBJECTS_PATH = dir('objects');
+process.env.THUMBNAILS_PATH = dir('thumbnails');
+process.env.OV_NETWORK_URL = 'https://openvibe.network';
+process.env.MEDIA_PUBLIC_URL = 'https://media.test';
+process.env.MEDIA_SIGNING_SECRET = 'test-signing-secret';
+process.env.MEDIA_OBJECT_MAX_MB = '1';
+process.env.MEDIA_PUBLIC_OBJECT_MAX_MB = '0'; // every public playback object violates here
+
+const db = require('../server/db/database');
+const auth = require('../server/auth');
+const model = require('../server/objects/model');
+const objectRoutes = require('../server/objects/routes');
+const { serviceAuth } = require('openvibe-contracts');
+const express = require('express');
+
+const keys = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });
+auth._setNetworkPublicKeyForTests(keys.publicKey);
+db.upsertApp({ app_id: 'live', api_key: 'live-key' });
+db.upsertApp({ app_id: 'games', api_key: 'games-key' });
+db.upsertApp({ app_id: 'tiny', api_key: 'tiny-key', quota_bytes: 20 });
+const now = Math.floor(Date.now() / 1000);
+const tok = (over = {}) => serviceAuth.signServiceToken({ iss: 'https://openvibe.network', sub: 'svc:example', actor_type: 'service', aud: ['openvibe.media'], cap: ['media.object.upload', 'media.object.read'], ns: ['live'], iat: now, exp: now + 300, jti: `tok_${crypto.randomBytes(6).toString('hex')}`, ...over }, keys.privateKey);
+const SUBJECT = 'usr_01JAB2C3D4E5F6G7H8J9K0MNPQ';
+
+// Same mount order as server/index.js.
+const app = express();
+app.put('/api/v2/:app/objects/:id/content', ...objectRoutes.contentHandlers);
+app.use(express.json());
+app.use('/api/v1/:app/files', require('../server/files/routes'));
+app.use('/api/v1/:app/vods', require('../server/vod/routes'));
+app.use('/api/v2/:app/objects', objectRoutes);
+app.use('/o', objectRoutes.publicRouter);
+const server = http.createServer(app);
+const sha = (b) => crypto.createHash('sha256').update(b).digest('hex');
+
+(async () => {
+ await new Promise((r) => server.listen(0, '127.0.0.1', r));
+ const base = `http://127.0.0.1:${server.address().port}`;
+ const local = (u) => base + String(u).replace('https://media.test', '');
+ const call = async (method, p, { bearer = 'live-key', body, headers = {}, raw } = {}) => {
+ const h = { ...headers };
+ if (bearer) h.authorization = `Bearer ${bearer}`;
+ let payload;
+ if (raw !== undefined) payload = raw;
+ else if (body !== undefined) { h['content-type'] = 'application/json'; payload = JSON.stringify(body); }
+ const res = await fetch(p.startsWith('http') ? p : base + p, { method, headers: h, body: payload, redirect: 'manual' });
+ const text = await res.text();
+ let json = null; try { json = JSON.parse(text); } catch { /* not json */ }
+ return { status: res.status, body: json, text, headers: res.headers };
+ };
+ const O = '/api/v2/live/objects';
+
+ // ── Init → PUT (upload token, no other credential) → complete ──
+ const hello = Buffer.from('hello world');
+ let r = await call('POST', O, { body: { kind: 'file', mime_type: 'text/plain', size_bytes: hello.length, filename: '../hello.txt', visibility: 'private', content_hash: sha(hello) } });
+ assert.strictEqual(r.status, 201, JSON.stringify(r.body));
+ const id = r.body.id;
+ assert.ok(/^med_[0-9A-HJKMNP-TV-Z]{26}$/.test(id));
+ assert.deepStrictEqual([r.body.object.lifecycle_status, r.body.object.metadata.filename, r.body.upload.method, r.body.upload.max_bytes], ['uploading', 'hello.txt', 'PUT', 11]);
+ const putUrl = local(r.body.upload.url);
+ r = await call('PUT', putUrl, { bearer: null, raw: Buffer.from('hello') });
+ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.object.size_mismatch']);
+ r = await call('PUT', putUrl.replace(/token=[^&]+/, 'token=1.forged'), { bearer: null, raw: hello });
+ assert.deepStrictEqual([r.status, r.body.code], [401, 'media.upload_token.invalid']);
+ r = await call('PUT', putUrl, { bearer: null, raw: hello, headers: { 'content-type': 'application/json' } });
+ assert.strictEqual(r.status, 200, JSON.stringify(r.body));
+ assert.deepStrictEqual([r.body.size_bytes, r.body.content_hash], [11, sha(hello)]);
+ const stored = model.objectFilePath(model.getObject(id));
+ assert.strictEqual(fs.readFileSync(stored, 'utf8'), 'hello world', 'bytes stored under OBJECTS_PATH/<app>/<id>');
+ r = await call('POST', `${O}/${id}/complete`);
+ assert.strictEqual(r.status, 200, JSON.stringify(r.body));
+ assert.deepStrictEqual([r.body.lifecycle_status, r.body.mime_type, r.body.content_hash, r.body.public_url], ['ready', 'text/plain', sha(hello), null]);
+ r = await call('GET', `${O}/${id}`);
+ assert.strictEqual(r.status, 200);
+ assert.deepStrictEqual(r.body.locations, [{ provider: 'local', storage_class: 'hot', state: 'present', size_bytes: 11, verified_at: r.body.locations[0].verified_at, canonical: true }]);
+ assert.ok(!r.text.includes(tmp), 'no server paths in responses');
+ console.log('✅ init → PUT (upload token) → complete; size + sha256 checked');
+
+ // ── Private download: signed, short-lived ──
+ r = await call('GET', `/o/${id}`, { bearer: null });
+ assert.strictEqual(r.status, 404, 'a private object is not reachable without a signature');
+ r = await call('GET', `${O}/${id}/download`);
+ assert.strictEqual(r.status, 200);
+ assert.strictEqual(r.body.public, false);
+ assert.ok(Date.parse(r.body.expires_at) > Date.now());
+ const signed = r.body.url;
+ r = await call('GET', local(signed), { bearer: null });
+ assert.deepStrictEqual([r.status, r.text, r.headers.get('content-type'), r.headers.get('x-content-type-options'), r.headers.get('cache-control')], [200, 'hello world', 'text/plain', 'nosniff', 'private, no-store']);
+ assert.strictEqual(r.headers.get('content-disposition'), 'inline; filename="hello.txt"');
+ r = await call('GET', local(signed).replace(/sig=[^&]+/, 'sig=AAAA'), { bearer: null });
+ assert.strictEqual(r.status, 404, 'tampered signature');
+ const past = now - 10;
+ const oldSig = crypto.createHmac('sha256', 'test-signing-secret').update(`get\n${id}\n${past}`).digest('base64url');
+ r = await call('GET', `/o/${id}?exp=${past}&sig=${oldSig}`, { bearer: null });
+ assert.strictEqual(r.status, 404, 'expired signature');
+ const putSig = crypto.createHmac('sha256', 'test-signing-secret').update(`put\n${id}\n${now + 100}`).digest('base64url');
+ r = await call('GET', `/o/${id}?exp=${now + 100}&sig=${putSig}`, { bearer: null });
+ assert.strictEqual(r.status, 404, 'an upload signature is not a download signature');
+ r = await call('GET', `${O}/${id}/download?redirect=1`);
+ assert.strictEqual(r.status, 302);
+ console.log('✅ private objects: signed short-lived URLs only');
+
+ // ── Public object: app-key PUT, 302 to /o, Range ──
+ const png = Buffer.concat([Buffer.from('\x89PNG\r\n\x1a\n', 'binary'), crypto.randomBytes(100)]);
+ r = await call('POST', O, { body: { kind: 'screenshot', visibility: 'public' } });
+ const pubId = r.body.id;
+ r = await call('PUT', `${O}/${pubId}/content`, { raw: png, headers: { 'content-type': 'image/png' } });
+ assert.strictEqual(r.status, 200, JSON.stringify(r.body));
+ r = await call('POST', `${O}/${pubId}/complete`);
+ assert.deepStrictEqual([r.body.mime_type, r.body.public_url], ['image/png', `https://media.test/o/${pubId}`]);
+ r = await call('GET', `${O}/${pubId}/download`);
+ assert.deepStrictEqual([r.status, r.headers.get('location')], [302, `https://media.test/o/${pubId}`]);
+ r = await call('GET', `/o/${pubId}`, { bearer: null, headers: { range: 'bytes=0-7' } });
+ assert.deepStrictEqual([r.status, r.headers.get('content-range'), r.headers.get('content-type')], [206, `bytes 0-7/${png.length}`, 'image/png']);
+ console.log('✅ public objects: 302 to the public location, ranged bytes at /o/:id');
+
+ // ── Service tokens ──
+ r = await call('POST', O, { bearer: tok(), headers: { 'x-ov-subject': `user:${SUBJECT}` }, body: { kind: 'asset', size_bytes: 3 } });
+ assert.strictEqual(r.status, 201, JSON.stringify(r.body));
+ assert.deepStrictEqual([r.body.object.owner.subject, r.body.object.metadata.created_by], [SUBJECT, 'svc:example']);
+ const svcId = r.body.id;
+ r = await call('PUT', `${O}/${svcId}/content`, { bearer: tok(), raw: Buffer.from('abc') });
+ assert.strictEqual(r.status, 200);
+ r = await call('POST', `${O}/${svcId}/complete`, { bearer: tok() });
+ assert.strictEqual(r.body.lifecycle_status, 'ready');
+ r = await call('POST', O, { bearer: tok(), headers: { 'x-ov-subject': 'alice' }, body: {} });
+ assert.deepStrictEqual([r.status, r.body.code], [400, 'media.object.invalid'], 'owner must be a usr_ subject');
+ r = await call('POST', O, { bearer: tok({ ns: ['community'] }), body: {} });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.namespace_denied']);
+ r = await call('POST', O, { bearer: tok({ cap: ['media.object.read'] }), body: {} });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'read grant cannot upload');
+ r = await call('GET', `${O}?owner=${SUBJECT}`, { bearer: tok({ cap: ['media.object.read'] }) });
+ assert.deepStrictEqual(r.body.objects.map(o => o.id), [svcId], 'read grant lists; owner filter');
+ r = await call('GET', `${O}/${svcId}`, { bearer: tok({ cap: ['media.object.upload'] }) });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'upload grant cannot read');
+ r = await call('POST', `${O}/${svcId}/holds`, { bearer: tok(), body: { kind: 'admin' } });
+ assert.deepStrictEqual([r.status, r.body.code], [403, 'capability.denied'], 'holds are app-key only');
+ r = await call('GET', `/api/v2/games/objects/${svcId}`, { bearer: 'games-key' });
+ assert.strictEqual(r.status, 404, 'objects are namespaced');
+ r = await call('GET', `/api/v2/games/objects/${svcId}`, { bearer: 'live-key' });
+ assert.strictEqual(r.status, 403, "another app's key is refused");
+ console.log('✅ service tokens: capability + namespace enforced, X-OV-Subject owns the object');
+
+ // ── Cursor listing ──
+ const page1 = await call('GET', `${O}?limit=2`);
+ assert.strictEqual(page1.body.objects.length, 2);
+ assert.ok(page1.body.next_cursor);
+ const page2 = await call('GET', `${O}?limit=2&cursor=${page1.body.next_cursor}`);
+ const seen = [...page1.body.objects, ...page2.body.objects].map(o => o.id);
+ assert.deepStrictEqual(seen, [svcId, pubId, id], 'newest first, no duplicates');
+ assert.strictEqual(page2.body.next_cursor, null);
+ assert.deepStrictEqual((await call('GET', `${O}?kind=screenshot`)).body.objects.map(o => o.id), [pubId]);
+ assert.deepStrictEqual((await call('GET', `${O}?visibility=private&kind=file`)).body.objects.map(o => o.id), [id]);
+ assert.strictEqual((await call('GET', `${O}?cursor=nope`)).status, 400);
+ console.log('✅ cursor listing with filters');
+
+ // ── Acting users see public + their own ──
+ r = await call('POST', O, { headers: { 'x-ov-user-id': '7' }, body: { kind: 'file', visibility: 'private' } });
+ const sevens = r.body.id;
+ assert.strictEqual(r.body.object.owner.user_id, 7);
+ assert.strictEqual((await call('GET', `${O}/${sevens}`, { headers: { 'x-ov-user-id': '8' } })).status, 404);
+ assert.ok(!(await call('GET', O, { headers: { 'x-ov-user-id': '8' } })).body.objects.some(o => o.id === sevens || o.id === id), 'no enumeration of others\' private objects');
+ assert.ok((await call('GET', O, { headers: { 'x-ov-user-id': '7' } })).body.objects.some(o => o.id === sevens));
+ assert.strictEqual((await call('DELETE', `${O}/${pubId}`, { headers: { 'x-ov-user-id': '8' } })).status, 403);
+ console.log('✅ acting users: public + own objects only');
+
+ // ── Holds, soft delete, restore ──
+ r = await call('POST', `${O}/${id}/holds`, { body: { kind: 'dmca', reason: 'notice 42' } });
+ assert.strictEqual(r.status, 201);
+ const holdId = r.body.id;
+ assert.strictEqual(r.body.created_by, 'app:live');
+ assert.strictEqual((await call('GET', `${O}/${id}`)).body.held, true);
+ r = await call('DELETE', `${O}/${id}`);
+ assert.deepStrictEqual([r.status, r.body.code], [409, 'media.object.held']);
+ assert.strictEqual((await call('POST', `${O}/${id}/holds`, { body: { kind: 'nope' } })).status, 400);
+ assert.strictEqual((await call('GET', `${O}/${id}/holds`)).body.holds.length, 1);
+ r = await call('DELETE', `${O}/${id}/holds/${holdId}`);
+ assert.ok(r.body.released_at);
+ assert.strictEqual((await call('GET', `${O}/${id}/holds`)).body.holds.length, 0);
+ assert.strictEqual((await call('GET', `${O}/${id}/holds?all=1`)).body.holds.length, 1);
+ r = await call('DELETE', `${O}/${id}`);
+ assert.deepStrictEqual([r.status, r.body.lifecycle_status], [200, 'deleted']);
+ assert.ok(fs.existsSync(stored), 'soft delete keeps the bytes');
+ assert.strictEqual((await call('GET', local(signed), { bearer: null })).status, 410, 'a deleted object is gone, even with a valid signature');
+ assert.strictEqual((await call('GET', `${O}/${id}/download`)).status, 410);
+ assert.ok(!(await call('GET', O)).body.objects.some(o => o.id === id), 'deleted objects leave the default listing');
+ assert.ok((await call('GET', `${O}?include_deleted=1`)).body.objects.some(o => o.id === id));
+ r = await call('POST', `${O}/${id}/restore`);
+ assert.deepStrictEqual([r.status, r.body.lifecycle_status], [200, 'ready']);
+ assert.strictEqual((await call('POST', `${O}/${id}/restore`)).status, 409);
+ console.log('✅ holds block deletion (409); soft delete keeps bytes; restore');
+
+ // ── Integrity + limits ──
+ r = await call('POST', O, { body: { kind: 'file', content_hash: sha(Buffer.from('expected')) } });
+ const badId = r.body.id;
+ await call('PUT', `${O}/${badId}/content`, { raw: Buffer.from('something else') });
+ r = await call('POST', `${O}/${badId}/complete`);
+ assert.deepStrictEqual([r.status, r.body.code], [422, 'media.object.hash_mismatch']);
+ assert.strictEqual((await call('POST', O, { body: { kind: 'file', content_hash: 'xyz' } })).status, 400);
+ assert.strictEqual((await call('POST', O, { body: { kind: 'nope' } })).status, 400);
+ r = await call('POST', O, { body: { kind: 'file', size_bytes: 2 * 1024 * 1024 } });
+ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.object.too_large'], 'MEDIA_OBJECT_MAX_MB at init');
+ r = await call('POST', O, { body: { kind: 'file' } });
+ r = await call('PUT', `${O}/${r.body.id}/content`, { raw: crypto.randomBytes(1024 * 1024 + 10) });
+ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.object.too_large'], 'and while streaming');
+ r = await call('POST', O, { body: { kind: 'vod', visibility: 'public', size_bytes: 10 } });
+ assert.deepStrictEqual([r.status, r.body.code], [422, 'media.invariant.public_object_too_large'], 'public playback objects over the invariant are refused');
+ r = await call('POST', O, { body: { kind: 'vod', visibility: 'private', size_bytes: 10 } });
+ assert.strictEqual(r.status, 201, 'private ones are not public playback');
+ const T = '/api/v2/tiny/objects';
+ assert.deepStrictEqual([(r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 30 } })).status, r.body.code], [413, 'media.quota.exceeded']);
+ assert.strictEqual((await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 10 } })).status, 201);
+ r = await call('POST', T, { bearer: 'tiny-key', body: { size_bytes: 15 } });
+ assert.strictEqual(r.status, 413, 'reservations count against the quota at init');
+ r = await call('POST', T, { bearer: 'tiny-key', body: {} });
+ r = await call('PUT', `${T}/${r.body.id}/content`, { bearer: 'tiny-key', raw: Buffer.alloc(15) });
+ assert.deepStrictEqual([r.status, r.body.code], [413, 'media.quota.exceeded'], 'and again with the real size');
+ assert.deepStrictEqual(fs.readdirSync(path.join(process.env.OBJECTS_PATH, '.tmp')), [], 'refused uploads leave no temp files');
+ console.log('✅ hash mismatch, size limit, invariant and quota enforced');
+
+ // ── Legacy projections through v2 ──
+ r = await call('POST', '/api/v1/live/vods', { body: { title: 'Rec', user_id: 3 } });
+ const vodId = r.body.id;
+ const legacy = `legacy:live:vod:${vodId}`;
+ r = await call('GET', `${O}/${legacy}`);
+ assert.deepStrictEqual([r.status, r.body.kind, r.body.lifecycle_status, r.body.legacy_ref], [200, 'vod', 'uploading', legacy], 'a v1 vod creates its object; readable by legacy ref');
+ const vodObj = r.body.id;
+ assert.strictEqual((await call('DELETE', `${O}/${vodObj}`)).body.code, 'media.object.legacy_managed');
+ fs.writeFileSync(path.join(process.env.VOD_PATH, 'v.webm'), 'x');
+ db.run('UPDATE vods SET file_path = ?, file_size = 1, duration_seconds = 5 WHERE id = ?', [path.join(process.env.VOD_PATH, 'v.webm'), vodId]);
+ model.sync('vod', vodId);
+ r = await call('GET', `${O}/${vodObj}/download`);
+ assert.deepStrictEqual([r.status, r.headers.get('location')], [302, `https://media.test/v/${vodId}`], 'public legacy objects redirect to their inherited URL');
+ await call('POST', `${O}/${vodObj}/holds`, { body: { kind: 'evidence' } });
+ r = await call('DELETE', `/api/v1/live/vods/${vodId}`);
+ assert.deepStrictEqual([r.status, r.body.code], [409, 'media.object.held'], 'v1 delete honours the hold');
+ assert.ok(fs.existsSync(path.join(process.env.VOD_PATH, 'v.webm')));
+ const fd = new FormData();
+ fd.append('file', new Blob([Buffer.from('data')], { type: 'text/plain' }), 'x.txt');
+ r = await fetch(`${base}/api/v1/live/files`, { method: 'POST', headers: { authorization: 'Bearer live-key' }, body: fd }).then(async (x) => ({ status: x.status, body: await x.json() }));
+ assert.strictEqual(r.status, 201);
+ const fileObj = (await call('GET', `${O}/legacy:live:file:${r.body.key}`)).body;
+ assert.deepStrictEqual([fileObj.kind, fileObj.content_hash, fileObj.visibility], ['file', r.body.sha256, 'public'], 'v1 file upload creates its object');
+ await call('POST', `${O}/${fileObj.id}/holds`, { body: { kind: 'moderation' } });
+ assert.strictEqual((await call('DELETE', `/api/v1/live/files/${r.body.key}`)).status, 409);
+ const v1List = await call('GET', '/api/v1/live/files');
+ assert.ok(!('object_id' in v1List.body.files[0]), 'v1 responses are unchanged');
+ console.log('✅ legacy projections: readable via v2, deleted only via v1, holds honoured there');
+
+ server.close();
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('object API v2: all checks passed');
+})().catch((err) => { console.error(err); server.close(); process.exit(1); });
new file mode 100644
new file mode 100644
@@ -0,0 +1,244 @@
@@ -0,0 +1,244 @@
+'use strict';
+// Canonical object model (roadmap Wave 4): the backfill projects every vod, clip, file, screenshot,
+// avatar and thumbnail row onto media_objects + media_locations without touching bytes or providers;
+// it is idempotent; the write hooks keep the model current; holds block deletion and tier moves on
+// every path; the public object-size invariant is recorded as policy.
+const assert = require('assert');
+const fs = require('fs');
+const os = require('os');
+const path = require('path');
+
+const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-objects-'));
+const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+process.env.DB_PATH = path.join(tmp, 'media.db');
+process.env.VOD_PATH = dir('vods');
+process.env.CLIPS_PATH = dir('clips');
+process.env.FILES_PATH = dir('files');
+process.env.THUMBNAILS_PATH = dir('thumbnails');
+process.env.PASTES_PATH = dir('pastes');
+process.env.OBJECTS_PATH = dir('objects');
+process.env.MEDIA_PUBLIC_URL = 'https://media.test';
+process.env.MEDIA_B2_BUCKET = 'bkt-b2'; // bucket name only: no endpoint, so B2 is not configured
+process.env.MEDIA_R2_BUCKET = 'bkt-r2';
+process.env.MEDIA_PUBLIC_OBJECT_TARGET_MB = '1';
+process.env.MEDIA_PUBLIC_OBJECT_WARN_MB = '2';
+process.env.MEDIA_PUBLIC_OBJECT_MAX_MB = '3';
+
+const contracts = require('openvibe-contracts');
+const db = require('../server/db/database');
+const model = require('../server/objects/model');
+const { backfill } = require('../server/objects/backfill');
+const invariant = require('../server/objects/invariant');
+const vodStorage = require('../server/vod/vod-storage');
+
+const MB = 1024 * 1024;
+const MED = /^med_[0-9A-HJKMNP-TV-Z]{26}$/;
+const d = db.getDb();
+const write = (p, bytes) => { fs.mkdirSync(path.dirname(p), { recursive: true }); fs.writeFileSync(p, Buffer.alloc(bytes, 7)); return p; };
+const objOf = (table, id) => model.getObject(d.prepare(`SELECT object_id FROM ${table} WHERE ${table === 'files' ? 'key' : 'id'} = ?`).get(id).object_id);
+const locs = (o) => Object.fromEntries(model.listLocations(o.id).map(l => [l.provider, l]));
+
+// ── Inherited rows, inserted raw (no hooks), the way the cutover import left them ──
+const vodFile = write(path.join(process.env.VOD_PATH, 'vod-live-1.webm'), 1000);
+write(path.join(process.env.THUMBNAILS_PATH, 'vod-1-111.jpg'), 50);
+const insVod = d.prepare(`INSERT INTO vods (id, app_id, user_id, title, file_path, file_size, duration_seconds, visibility, is_public, storage_provider, storage_key, thumbnail_url, clips_only, is_recording, created_at)
+ VALUES (@id, @app, @user, @title, @file, @size, @dur, @vis, @pub, @prov, @key, @thumb, @co, 0, @at)`);
+const V = (o) => insVod.run({ app: 'live', user: 5, title: 'V', file: null, size: 0, dur: 60, vis: 'public', pub: 1, prov: 'local', key: null, thumb: null, co: 0, at: '2025-06-01 10:00:00', ...o });
+V({ id: 1, file: vodFile, size: 1000, thumb: '/t/vod-1-111.jpg', at: '2025-01-01 00:00:00' }); // local, present
+V({ id: 2, file: '/old/host/vods/old.webm', size: 5 * MB, vis: 'unlisted', pub: 0, prov: 'b2', key: 'vods/old.webm' }); // offloaded
+V({ id: 3, file: '/old/host/vods/pop.mp4', size: 2.5 * MB, prov: 'r2', key: 'vods/pop.mp4', thumb: 'https://elsewhere.example/x.jpg' });
+V({ id: 4, file: '/opt/hobostreamer/data/vods/gone.webm', size: 50 }); // local but gone
+V({ id: 5, file: path.join(process.env.VOD_PATH, 'eph.webm'), co: 1 }); // clips-only
+V({ id: 6, dur: 0 }); // ghost: no file yet
+V({ id: 7, app: 'games', file: vodFile, size: 1000, vis: 'private', pub: 0 });
+const clipFile = write(path.join(process.env.CLIPS_PATH, 'clip-1.webm'), 300);
+d.prepare("INSERT INTO clips (id, app_id, vod_id, user_id, title, file_path, status, visibility, is_public, start_time, end_time) VALUES (1, 'live', 1, 9, 'C', ?, 'ready', 'public', 1, 5, 15)").run(clipFile);
+d.prepare("INSERT INTO clips (id, app_id, vod_id, user_id, title, file_path, status) VALUES (2, 'live', 1, 9, 'C2', '', 'processing')").run();
+write(path.join(process.env.FILES_PATH, 'live', 'abc123-a.txt'), 12);
+d.prepare("INSERT INTO files (key, app_id, user_id, original_name, size, mime, sha256) VALUES ('abc123-a.txt', 'live', 3, 'a.txt', 12, 'text/plain', ?)").run('a'.repeat(64));
+const shot = write(path.join(process.env.PASTES_PATH, 'screenshots', 's1.png'), 40);
+const avatar = write(path.join(process.env.PASTES_PATH, 'screenshots', 'avatar-n4.webp'), 30);
+d.prepare("INSERT INTO pastes (id, app_id, slug, user_id, type, title, content, visibility, screenshot_path, metadata) VALUES (1, 'live', 'shot-one', 2, 'screenshot', 'S', '', 'public', ?, NULL)").run(shot);
+d.prepare("INSERT INTO pastes (id, app_id, slug, user_id, type, title, content, visibility, screenshot_path, metadata) VALUES (2, 'network', 'avatar-four', 4, 'screenshot', 'Avatar', '', 'unlisted', ?, ?)").run(avatar, JSON.stringify({ kind: 'avatar', mime_type: 'image/webp' }));
+d.prepare("INSERT INTO pastes (id, app_id, slug, type, title, content, visibility) VALUES (3, 'live', 'text-three', 'paste', 'T', 'hello', 'public')").run();
+d.prepare("INSERT INTO pastes (id, app_id, slug, type, title, content, visibility, screenshot_path) VALUES (4, 'live', 'shot-nopath', 'screenshot', 'S', '', 'public', NULL)").run();
+const mtimes = [vodFile, clipFile, shot, avatar].map(p => fs.statSync(p).mtimeMs);
+
+// ── Dry run: a full report, nothing written ──
+const dry = backfill({ dryRun: true });
+assert.strictEqual(dry.dry_run, true);
+assert.deepStrictEqual(dry.counts.vod, { seen: 7, created: 6, updated: 0, skipped: 1 });
+assert.strictEqual(d.prepare('SELECT COUNT(*) c FROM media_objects').get().c, 0, 'dry run writes no objects');
+assert.strictEqual(d.prepare('SELECT COUNT(*) c FROM vods WHERE object_id IS NOT NULL').get().c, 0, 'dry run links no rows');
+assert.strictEqual(model.parseJson(d.prepare("SELECT value FROM media_settings WHERE key = 'objects.backfill.last_report'").get()?.value, null), null, 'dry run stores no report');
+console.log('✅ dry run reports without writing');
+
+// ── Real run ──
+const r = backfill();
+assert.deepStrictEqual(r.counts.vod, { seen: 7, created: 6, updated: 0, skipped: 1 });
+assert.deepStrictEqual(r.counts.clip, { seen: 2, created: 2, updated: 0, skipped: 0 });
+assert.deepStrictEqual(r.counts.file, { seen: 1, created: 1, updated: 0, skipped: 0 });
+assert.deepStrictEqual(r.counts.screenshot, { seen: 2, created: 1, updated: 0, skipped: 1 });
+assert.deepStrictEqual(r.counts.avatar, { seen: 1, created: 1, updated: 0, skipped: 0 });
+assert.deepStrictEqual(r.counts.thumbnail, { seen: 2, created: 1, updated: 0, skipped: 1 });
+assert.deepStrictEqual(r.skipped.map(s => s.reason).sort(), ['clips-only recording (ephemeral, never published)', 'external thumbnail url', 'screenshot paste without a file path']);
+assert.deepStrictEqual(r.errors, []);
+assert.ok(d.prepare("SELECT value FROM media_settings WHERE key = 'objects.backfill.last_report'").get(), 'report recorded');
+assert.deepStrictEqual([vodFile, clipFile, shot, avatar].map(p => fs.statSync(p).mtimeMs), mtimes, 'no bytes touched');
+
+const v1 = objOf('vods', 1), v2 = objOf('vods', 2), v3 = objOf('vods', 3), v4 = objOf('vods', 4), v6 = objOf('vods', 6), v7 = objOf('vods', 7);
+for (const o of [v1, v2, v3, v4, v6, v7]) {
+ assert.ok(MED.test(o.id), o.id);
+ assert.ok(contracts.validate('media.media-ref@1', { media_id: o.id }).valid, 'canonical id is a valid media-ref');
+ assert.ok(contracts.validate('media.media-ref@1', { media_id: o.legacy_ref }).valid, `legacy ref valid: ${o.legacy_ref}`);
+}
+assert.ok(v1.id < v2.id, 'ids sort by the original creation time');
+assert.strictEqual(v1.legacy_ref, 'legacy:live:vod:1');
+assert.deepStrictEqual([v1.kind, v1.visibility, v1.lifecycle_status, v1.size_bytes, v1.owner_app, v1.owner_user_id, v1.mime_type], ['vod', 'public', 'ready', 1000, 'live', 5, 'video/webm']);
+assert.strictEqual(locs(v1).local.state, 'present');
+assert.strictEqual(locs(v1).local.key, vodFile);
+assert.ok(locs(v1).local.verified_at, 'a local file that exists is verified present');
+assert.deepStrictEqual([v1.canonical_provider, v1.canonical_key], ['local', vodFile]);
+assert.deepStrictEqual(Object.keys(locs(v2)), ['b2'], 'offloaded vod: B2 copy only');
+assert.deepStrictEqual([locs(v2).b2.state, locs(v2).b2.bucket, locs(v2).b2.key, locs(v2).b2.storage_class, v2.canonical_provider], ['pending', 'bkt-b2', 'vods/old.webm', 'cold', 'b2']);
+assert.deepStrictEqual(Object.keys(locs(v3)).sort(), ['b2', 'r2'], 'R2 cache implies the B2 canonical');
+assert.deepStrictEqual([locs(v3).r2.state, locs(v3).r2.storage_class, v3.canonical_provider], ['pending', 'cache', 'b2']);
+assert.strictEqual(locs(v4).local.state, 'missing', 'a local file that is not on disk is recorded missing');
+assert.strictEqual(model.listLocations(v6.id).length, 0);
+assert.strictEqual(v6.lifecycle_status, 'uploading');
+assert.deepStrictEqual([v7.app_id, v7.namespace, v7.visibility], ['games', 'games', 'private']);
+assert.strictEqual(d.prepare('SELECT object_id FROM vods WHERE id = 5').get().object_id, null, 'clips-only row stays unprojected');
+
+const c1 = objOf('clips', 1), c2 = objOf('clips', 2);
+assert.deepStrictEqual([c1.kind, c1.lifecycle_status, c1.size_bytes, locs(c1).local.state], ['clip', 'ready', 300, 'present']);
+assert.strictEqual(c2.lifecycle_status, 'uploading', 'a clip still being cut is uploading');
+const rel = d.prepare("SELECT * FROM media_relationships WHERE from_object_id = ? AND relation = 'clip_of'").get(c1.id);
+assert.strictEqual(rel.to_object_id, v1.id, 'clip_of links the clip to its vod');
+assert.deepStrictEqual(JSON.parse(rel.metadata), { start_time: 5, end_time: 15 });
+
+const f = objOf('files', 'abc123-a.txt');
+assert.deepStrictEqual([f.kind, f.visibility, f.content_hash, f.mime_type, f.legacy_ref, locs(f).local.checksum], ['file', 'public', 'a'.repeat(64), 'text/plain', 'legacy:live:file:abc123-a.txt', 'a'.repeat(64)]);
+const s1 = objOf('pastes', 1), av = objOf('pastes', 2);
+assert.deepStrictEqual([s1.kind, s1.legacy_ref, s1.size_bytes], ['screenshot', 'legacy:live:paste:shot-one', 40]);
+assert.deepStrictEqual([av.kind, av.app_id, av.legacy_ref, av.visibility, av.mime_type], ['avatar', 'network', 'legacy:network:avatar:avatar-four', 'unlisted', 'image/webp']);
+assert.strictEqual(d.prepare('SELECT object_id FROM pastes WHERE id = 3').get().object_id, null, 'text pastes have no bytes object');
+
+const tv = model.getVariant(v1.id, 'thumbnail');
+const th = model.getObject(tv.derived_object_id);
+assert.deepStrictEqual([th.kind, th.legacy_ref, th.visibility, locs(th).local.state], ['thumbnail', 'legacy:live:thumbnail:vod-1-111.jpg', 'public', 'present']);
+assert.ok(d.prepare("SELECT 1 FROM media_relationships WHERE from_object_id = ? AND relation = 'thumbnail_of' AND to_object_id = ?").get(th.id, v1.id));
+console.log('✅ backfill: one object per row, locations mirror where the bytes are, relationships + thumbnail variants');
+
+// ── Idempotent ──
+const countAll = () => ['media_objects', 'media_locations', 'media_relationships', 'media_variants'].map(t => d.prepare(`SELECT COUNT(*) c FROM ${t}`).get().c);
+const before = countAll();
+const again = backfill();
+assert.strictEqual(again.totals.created, 0, 'a re-run creates nothing');
+assert.strictEqual(again.counts.vod.updated, 6);
+assert.deepStrictEqual(countAll(), before);
+assert.strictEqual(objOf('vods', 1).id, v1.id, 'ids are stable');
+assert.strictEqual(backfill({ onlyMissing: true }).totals.created, 0, 'only-missing finds nothing new');
+console.log('✅ backfill is idempotent');
+
+// ── Remote knowledge survives a re-projection ──
+model.setLocationState(locs(v2).b2.id, { state: 'present', size_bytes: 5 * MB });
+model.sync('vod', 2);
+assert.strictEqual(locs(v2).b2.state, 'present', 'a verified remote copy is not downgraded to pending');
+console.log('✅ verified remote state is kept across syncs');
+
+// ── Write hooks on the inherited APIs ──
+write(path.join(process.env.FILES_PATH, 'live', 'def456-b.png'), 20);
+db.createFile({ key: 'def456-b.png', app_id: 'live', user_id: 3, original_name: 'b.png', size: 20, mime: 'image/png', sha256: 'b'.repeat(64) });
+assert.strictEqual(objOf('files', 'def456-b.png').content_hash, 'b'.repeat(64), 'file upload creates its object');
+db.setVodVisibility(1, 'private');
+assert.strictEqual(model.getObject(v1.id).visibility, 'private', 'visibility follows');
+assert.strictEqual(model.getObject(th.id).visibility, 'unlisted', 'a private vod thumbnail is still reachable by name: unlisted');
+// Thumbnail regenerated: same object, new file.
+write(path.join(process.env.THUMBNAILS_PATH, 'vod-1-222.jpg'), 60);
+db.run("UPDATE vods SET thumbnail_url = '/t/vod-1-222.jpg' WHERE id = 1");
+model.sync('vod', 1);
+const th2 = model.getObject(model.getVariant(v1.id, 'thumbnail').derived_object_id);
+assert.strictEqual(th2.id, th.id, 'one thumbnail object per parent, updated in place');
+assert.deepStrictEqual([th2.legacy_ref, th2.size_bytes], ['legacy:live:thumbnail:vod-1-222.jpg', 60]);
+const newClip = db.createClip({ app_id: 'live', vod_id: 1, user_id: 9, title: 'N', file_path: clipFile, status: 'ready' }).lastInsertRowid;
+assert.strictEqual(objOf('clips', newClip).kind, 'clip', 'clip creation creates its object');
+db.run('DELETE FROM clips WHERE id = ?', [newClip]);
+const deadClip = d.prepare('SELECT * FROM media_objects WHERE legacy_ref = ?').get(`legacy:live:clip:${newClip}`);
+assert.strictEqual(deadClip.lifecycle_status, 'deleted', 'deleting the row marks its object deleted (trigger — every delete path)');
+assert.ok(deadClip.deleted_at);
+console.log('✅ inherited writes keep the model current');
+
+// ── Tier move re-projection ──
+const v4row = d.prepare('SELECT * FROM vods WHERE id = 4').get();
+db.run("UPDATE vods SET storage_provider = 'b2', storage_key = 'vods/gone.webm' WHERE id = 4");
+model.afterTierMove(4, ['b2']);
+assert.deepStrictEqual(Object.keys(locs(v4)), ['b2'], 'after offload: the local row goes, the B2 copy appears');
+assert.strictEqual(locs(v4).b2.state, 'present', 'the copy the move verified is present');
+assert.ok(locs(v4).b2.verified_at);
+assert.ok(v4row);
+console.log('✅ tier moves update media_locations');
+
+(async () => {
+ // ── Retention holds ──
+ const hold = model.placeHold({ object_id: v1.id, kind: 'dmca', reason: 'notice #1', created_by: 'admin:alex' });
+ assert.strictEqual(model.isHeld(v1.id), true);
+ assert.throws(() => db.run('DELETE FROM vods WHERE id = 1'), /retention hold/, 'a held row cannot be deleted by any path');
+ assert.ok(d.prepare('SELECT 1 FROM vods WHERE id = 1').get(), 'row still there');
+ assert.throws(() => db.run("UPDATE media_objects SET lifecycle_status = 'deleted' WHERE id = ?", [v1.id]), /retention hold/);
+ await vodStorage.deleteVodObjects(d.prepare('SELECT * FROM vods WHERE id = 1').get());
+ assert.ok(fs.existsSync(vodFile), 'held bytes are not deleted');
+ const cold = await vodStorage.moveToCold(1);
+ assert.deepStrictEqual([cold.ok, cold.held], [false, true], 'held objects are not demoted to B2');
+ const prom = await vodStorage.promoteToR2(1);
+ assert.strictEqual(prom.held, true, 'nor promoted');
+ db.run("UPDATE vods SET storage_provider = 'r2' WHERE id = 1");
+ assert.strictEqual((await vodStorage.demoteFromR2(1)).held, true, 'nor demoted from R2');
+ db.run("UPDATE vods SET storage_provider = 'local' WHERE id = 1");
+ assert.throws(() => model.placeHold({ object_id: v1.id, kind: 'whim' }), /hold kind/);
+ model.releaseHold(hold.id, 'admin:alex');
+ assert.strictEqual(model.isHeld(v1.id), false);
+ assert.strictEqual(model.listHolds(v1.id, { includeReleased: true })[0].released_by, 'admin:alex');
+ console.log('✅ holds block deletion and tier moves until released');
+
+ // ── Invariant (thresholds from env: target 1, warn 2, max 3 MB) ──
+ assert.strictEqual(invariant.classify(0.5 * MB), 'ok');
+ assert.strictEqual(invariant.classify(1.5 * MB), 'above_target');
+ assert.strictEqual(invariant.classify(2.5 * MB), 'warn');
+ assert.strictEqual(invariant.classify(5 * MB), 'violation');
+ assert.strictEqual(invariant.wouldViolate({ kind: 'vod', visibility: 'private', size_bytes: 9 * MB }), false, 'private objects are not public playback');
+ assert.strictEqual(invariant.wouldViolate({ kind: 'file', visibility: 'public', size_bytes: 9 * MB }), false, 'only playback kinds');
+ const rows = () => d.prepare('SELECT o.legacy_ref, v.level, v.resolved_at FROM media_invariant_violations v JOIN media_objects o ON o.id = v.object_id ORDER BY o.legacy_ref').all();
+ assert.deepStrictEqual(rows().map(x => [x.legacy_ref, x.level, x.resolved_at]), [['legacy:live:vod:2', 'violation', null], ['legacy:live:vod:3', 'warn', null]], 'sync records public playback objects over warn/max');
+ const scan = invariant.scan({ record: false });
+ assert.deepStrictEqual(scan.counts, { ok: 2, above_target: 0, warn: 1, violation: 1 }); // ok: clip 1, vod 4 (vod 1 is private now)
+ db.setVodVisibility(3, 'private');
+ assert.ok(rows().find(x => x.legacy_ref === 'legacy:live:vod:3').resolved_at, 'making it private resolves the row');
+ console.log('✅ public object-size invariant is policy: classified, recorded, resolved');
+
+ // ── Native soft delete + purge ──
+ const nid = model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: 'ready', visibility: 'private', size_bytes: 4 });
+ const npath = write(model.objectFilePath(model.getObject(nid)), 4);
+ model.upsertLocation(nid, { provider: 'local', key: npath, state: 'present', size_bytes: 4 });
+ const heldId = model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: 'ready', size_bytes: 4 });
+ const hpath = write(model.objectFilePath(model.getObject(heldId)), 4);
+ model.upsertLocation(heldId, { provider: 'local', key: hpath, state: 'present' });
+ const deleted = model.softDelete(model.getObject(nid), { by: 'test' });
+ assert.strictEqual(deleted.lifecycle_status, 'deleted');
+ assert.ok(model.parseJson(deleted.metadata).retention_until, 'retention recorded');
+ assert.ok(fs.existsSync(npath), 'soft delete keeps the bytes');
+ assert.strictEqual(model.purgeExpired({ retentionDays: 1 }), 0, 'inside the retention period nothing is purged');
+ const h2 = model.placeHold({ object_id: heldId, kind: 'evidence' });
+ assert.throws(() => model.softDelete(model.getObject(heldId)), (e) => e.code === 'media.object.held');
+ model.releaseHold(h2.id);
+ model.softDelete(model.getObject(heldId));
+ model.placeHold({ object_id: heldId, kind: 'evidence' }); // held after deletion: still kept
+ assert.strictEqual(model.purgeExpired({ retentionDays: 0 }), 1);
+ assert.ok(!fs.existsSync(npath), 'expired native bytes are purged');
+ assert.ok(fs.existsSync(hpath), 'held bytes survive the purge');
+ assert.ok(model.parseJson(model.getObject(nid).metadata).purged_at);
+ assert.strictEqual(model.restore(model.getObject(nid)), null, 'a purged object cannot be restored');
+ console.log('✅ native soft delete keeps bytes for the retention period; purge skips held objects');
+
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('objects model + backfill: all checks passed');
+})().catch((err) => { console.error(err); process.exit(1); });
new file mode 100644
new file mode 100644
@@ -0,0 +1,142 @@
@@ -0,0 +1,142 @@
+'use strict';
+// Reconciliation of the object model against the bytes, with a fake B2/R2 provider: read-only by
+// default, --verify records what the HEADs found; it detects missing canonical copies, lost local
+// files, size/hash mismatches, orphan locations, deleted-but-still-served objects and unprojected
+// rows. Plus the three scripts end to end (backfill --dry-run, reconcile, invariant).
+const assert = require('assert');
+const fs = require('fs');
+const os = require('os');
+const path = require('path');
+const crypto = require('crypto');
+const { execFileSync } = require('child_process');
+
+const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-media-reconcile-'));
+const dir = (n) => { const d = path.join(tmp, n); fs.mkdirSync(d, { recursive: true }); return d; };
+const env = {
+ DB_PATH: path.join(tmp, 'media.db'), VOD_PATH: dir('vods'), CLIPS_PATH: dir('clips'), FILES_PATH: dir('files'),
+ THUMBNAILS_PATH: dir('thumbnails'), PASTES_PATH: dir('pastes'), OBJECTS_PATH: dir('objects'), MEDIA_PUBLIC_URL: 'https://media.test',
+ MEDIA_PUBLIC_OBJECT_TARGET_MB: '1', MEDIA_PUBLIC_OBJECT_WARN_MB: '2', MEDIA_PUBLIC_OBJECT_MAX_MB: '3',
+};
+Object.assign(process.env, env);
+
+const db = require('../server/db/database');
+const model = require('../server/objects/model');
+const { reconcile, summarize } = require('../server/objects/reconcile');
+
+const MB = 1024 * 1024;
+const d = db.getDb();
+const write = (p, content) => { fs.mkdirSync(path.dirname(p), { recursive: true }); fs.writeFileSync(p, content); return p; };
+const V = d.prepare(`INSERT INTO vods (id, app_id, file_path, file_size, duration_seconds, visibility, is_public, storage_provider, storage_key)
+ VALUES (?, 'live', ?, ?, 60, 'public', 1, ?, ?)`);
+
+// Rows: a healthy local vod; an offloaded one whose B2 copy is fine; one whose B2 copy is gone but
+// R2 still has it; one whose B2 copy has the wrong size; a local vod whose file vanished.
+V.run(1, write(path.join(env.VOD_PATH, 'a.webm'), Buffer.alloc(100)), 100, 'local', null);
+V.run(2, '/gone/b.webm', 200, 'b2', 'vods/b.webm');
+V.run(3, '/gone/c.webm', 300, 'r2', 'vods/c.webm');
+V.run(4, '/gone/d.webm', 400, 'b2', 'vods/d.webm');
+V.run(5, write(path.join(env.VOD_PATH, 'e.webm'), Buffer.alloc(10)), 10, 'local', null);
+const fileBytes = Buffer.from('file-bytes');
+write(path.join(env.FILES_PATH, 'live', 'k1-f.bin'), fileBytes);
+d.prepare("INSERT INTO files (key, app_id, size, mime, sha256) VALUES ('k1-f.bin', 'live', ?, 'application/octet-stream', ?)").run(fileBytes.length, crypto.createHash('sha256').update(fileBytes).digest('hex'));
+
+// Scripts: a dry-run backfill reports and writes nothing.
+const run = (script, args = []) => {
+ try { return { code: 0, out: execFileSync(process.execPath, [path.join(__dirname, '..', 'scripts', script), ...args], { env: { ...process.env, ...env }, encoding: 'utf8' }) }; }
+ catch (e) { return { code: e.status, out: String(e.stdout) }; }
+};
+const dry = JSON.parse(run('backfill-objects.js', ['--dry-run', '--json']).out);
+assert.deepStrictEqual([dry.dry_run, dry.counts.vod.created, dry.counts.file.created], [true, 5, 1]);
+assert.strictEqual(d.prepare('SELECT COUNT(*) c FROM media_objects').get().c, 0, 'dry-run script wrote nothing');
+const real = run('backfill-objects.js');
+assert.strictEqual(real.code, 0, real.out);
+assert.match(real.out, /vod 5\+\/0~\/0 skipped/);
+assert.strictEqual(d.prepare('SELECT COUNT(*) c FROM media_objects').get().c, 6);
+console.log('✅ backfill script: --dry-run reports only; a real run projects');
+
+fs.unlinkSync(path.join(env.VOD_PATH, 'e.webm')); // vod 5 loses its file after backfill
+const objId = (vodId) => d.prepare('SELECT object_id FROM vods WHERE id = ?').get(vodId).object_id;
+// Anomalies the reconciler must notice:
+d.prepare("INSERT INTO media_locations (object_id, provider, key, state) VALUES ('med_01JAB2C3D4E5F6G7H8J9K0MNPQ', 'local', '/nowhere', 'present')").run(); // orphan location
+db.run("UPDATE media_objects SET size_bytes = 999 WHERE id = ?", [objId(1)]); // local size mismatch
+db.run("UPDATE media_objects SET content_hash = ? WHERE legacy_ref = 'legacy:live:file:k1-f.bin'", ['0'.repeat(64)]); // hash mismatch
+db.run("UPDATE media_objects SET lifecycle_status = 'deleted' WHERE id = ?", [objId(2)]); // deleted, but /v/2 still serves it
+d.prepare("INSERT INTO clips (id, app_id, file_path, status) VALUES (9, 'live', '/x.webm', 'ready')").run(); // row with no object
+const ready = model.createObject({ app_id: 'live', kind: 'file', lifecycle_status: 'ready' }); // ready object, no location
+
+// Fake provider: key → size (absent = 404). B2 lost c.webm (R2 still has it) and holds a wrong-size d.webm.
+const bucket = { b2: { 'vods/b.webm': 200, 'vods/d.webm': 12345 }, r2: { 'vods/c.webm': 300 } };
+const heads = [];
+const fakeHead = async (provider, key) => { heads.push(`${provider}:${key}`); const s = bucket[provider][key]; return s === undefined ? null : { size: s, etag: 'x' }; };
+
+(async () => {
+ const snapshot = () => JSON.stringify(d.prepare('SELECT * FROM media_locations ORDER BY id').all());
+ const before = snapshot();
+ const ro = await reconcile({ head: fakeHead, hash: true });
+ assert.strictEqual(snapshot(), before, 'default mode writes nothing');
+ assert.strictEqual(heads.length, 0, 'default mode calls no provider');
+ const count = (rep, k) => rep.issues[k].count;
+ assert.strictEqual(count(ro, 'orphan_location'), 1);
+ assert.strictEqual(count(ro, 'missing_local_file'), 1);
+ assert.strictEqual(ro.issues.missing_local_file.items[0].legacy_ref, 'legacy:live:vod:5');
+ assert.strictEqual(count(ro, 'size_mismatch'), 1, 'local size mismatch');
+ assert.strictEqual(count(ro, 'hash_mismatch'), 1);
+ assert.strictEqual(count(ro, 'deleted_publicly_reachable'), 1);
+ assert.strictEqual(ro.issues.deleted_publicly_reachable.items[0].table, 'vods');
+ assert.strictEqual(count(ro, 'missing_projection'), 1);
+ assert.deepStrictEqual(ro.issues.missing_projection.items[0], { table: 'clips', id: 9 });
+ assert.strictEqual(count(ro, 'no_canonical_location'), 1);
+ assert.strictEqual(ro.issues.no_canonical_location.items[0].object_id, ready);
+ assert.strictEqual(count(ro, 'canonical_missing_replica_present'), 0, 'remote copies are unverified until --verify');
+ assert.strictEqual(ro.counts.remote_unverified, 4, 'b2 ×3 + r2 ×1 pending');
+ assert.ok(summarize(ro).includes('orphan_location: 1'));
+ console.log('✅ read-only pass: local checks, orphans, projections, deleted-but-served — no writes, no provider calls');
+
+ const v = await reconcile({ verify: true, head: fakeHead });
+ assert.strictEqual(heads.length, 4, 'every remote location HEADed once');
+ assert.strictEqual(count(v, 'remote_missing'), 1, 'vod 3 canonical B2 copy is gone');
+ assert.strictEqual(v.issues.remote_missing.items[0].key, 'vods/c.webm');
+ assert.strictEqual(count(v, 'canonical_missing_replica_present'), 1);
+ assert.deepStrictEqual(v.issues.canonical_missing_replica_present.items[0].present, ['r2'], 'the R2 cache still has it');
+ assert.strictEqual(count(v, 'size_mismatch'), 2, 'local vod 1 + remote vod 4');
+ assert.strictEqual(count(v, 'no_present_copy'), 2, 'vod 4 (only copy corrupt) and vod 5 (only copy missing)');
+ const st = (vodId, p) => d.prepare('SELECT state, verified_at, size_bytes FROM media_locations WHERE object_id = ? AND provider = ?').get(objId(vodId), p);
+ assert.strictEqual(st(3, 'b2').state, 'missing');
+ assert.strictEqual(st(3, 'r2').state, 'present');
+ assert.strictEqual(st(4, 'b2').state, 'corrupt');
+ assert.strictEqual(st(4, 'b2').size_bytes, 12345);
+ assert.strictEqual(st(5, 'local').state, 'missing');
+ assert.ok(st(3, 'r2').verified_at, '--verify records verified_at');
+ assert.ok(v.counts.locations_updated >= 6);
+ console.log('✅ --verify: HEADs remotes (fake provider), records states; detects lost canonical with a surviving replica');
+
+ // An unconfigured provider is "unverifiable", never "missing".
+ const u = await reconcile({ verify: true, head: async () => undefined });
+ assert.strictEqual(u.counts.remote_unverifiable, 4);
+ assert.strictEqual(count(u, 'remote_missing'), 0);
+
+ // The scripts against the same database.
+ const rc = run('reconcile-objects.js', ['--json']);
+ assert.strictEqual(rc.code, 1, 'issues → exit 1');
+ assert.ok(JSON.parse(rc.out).issues.orphan_location.count === 1);
+ // A deleted object's thumbnail that is still served is reachable too.
+ write(path.join(env.THUMBNAILS_PATH, 'vod-1-5.jpg'), 'jpg');
+ db.run("UPDATE vods SET thumbnail_url = '/t/vod-1-5.jpg' WHERE id = 1");
+ model.sync('vod', 1);
+ db.run('DELETE FROM vods WHERE id = 1');
+ const t = await reconcile({});
+ assert.ok(t.issues.deleted_publicly_reachable.items.some(i => i.reason.startsWith('derivative')), 'thumbnail of a deleted vod is flagged');
+
+ db.run('UPDATE vods SET file_size = ? WHERE id = 3', [5 * MB]);
+ model.sync('vod', 3);
+ const inv = run('object-invariant.js', ['--json']);
+ const invReport = JSON.parse(inv.out);
+ assert.strictEqual(invReport.counts.violation, 1);
+ assert.strictEqual(invReport.objects[0].legacy_ref, 'legacy:live:vod:3');
+ assert.strictEqual(d.prepare("SELECT level FROM media_invariant_violations WHERE object_id = ? AND resolved_at IS NULL").get(objId(3)).level, 'violation');
+ assert.match(run('object-invariant.js', ['--dry-run']).out, /violation 1/);
+ console.log('✅ scripts: reconcile exit codes + JSON, invariant report records violations');
+
+ fs.rmSync(tmp, { recursive: true, force: true });
+ console.log('objects reconciliation: all checks passed');
+})().catch((err) => { console.error(err); process.exit(1); });