openvibers/openvibe.media README.md
README.md
| 1 | # OpenVibe.Media |
| 2 | |
| 3 | **Live site:** https://openvibe.media · **API docs:** https://openvibe.services · **For AI:** https://openvibe.media/llms.txt |
| 4 | **Run it:** `npm ci && npm start` · **Tests:** `npm test` |
| 5 | |
| 6 | ## Purpose |
| 7 | |
| 8 | Standalone multi-tenant media service (port **4100**) extracted from the |
| 9 | OpenVibe.Live predecessor. Owns VOD ingest/recording, storage tiering |
| 10 | (local → Backblaze B2 → Cloudflare R2), clips, generic files, the canonical |
| 11 | object model and thumbnails, for all OpenVibe apps (`live`, `games`, `tools`, |
| 12 | `network`, `community`) and developer projects. Pastes moved to OpenVibe.Community |
| 13 | on 2026-09-22; Media's read-only paste API is gone (T10 step 2), `/p/:slug` only |
| 14 | redirects there and legacy screenshots redirect to media objects ([Pastes](#pastes)). |
| 15 | |
| 16 | Implements **Media API v1** from `../CONTRACTS.md`. |
| 17 | |
| 18 | ``` |
| 19 | npm install |
| 20 | cp .env.example .env # the first-party tenants are created at boot (token-only) |
| 21 | npm start # node server/index.js |
| 22 | ``` |
| 23 | |
| 24 | Requires Node ≥ 20, `ffmpeg`/`ffprobe` on PATH. PostgreSQL 18 and Valkey 9 (OpenVibe.Host `roles/data/`, ADR-035); |
| 25 | without `DATABASE_URL`, development uses an embedded PGlite database in `data/pglite`. |
| 26 | |
| 27 | ## Owns |
| 28 | |
| 29 | - VODs (recording from RTMP or RTP, chunk ingest, finalize), clips (cutting), thumbnails, generic files |
| 30 | and avatars, in one PostgreSQL database (`ov_media` on the host's data role; schema in [migrations/](migrations/)) |
| 31 | and the local media directories |
| 32 | - the canonical object model (`/api/v2`: objects, locations, variants, holds, derivatives, lifecycle, |
| 33 | namespaces) and its jobs (`media_jobs`) |
| 34 | - storage tiering (local, Backblaze B2, Cloudflare R2), copy verification and the disk guardian |
| 35 | - tenants (`apps`), developer-project tenants, public serving (`/v`, `/c`, `/t`, `/f`, `/o`) and the |
| 36 | object explorer (`/me`) |
| 37 | - the `media.*` events and Media's Search documents |
| 38 | |
| 39 | ## Does not own |
| 40 | |
| 41 | - pastes since 2026-09-22 (OpenVibe.Community; Media keeps legacy screenshot files through media objects) |
| 42 | - channels, streams and who a VOD belongs to (OpenVibe.Live answers lineage), identity (OpenVibe.Network) |
| 43 | - AI analysis of recordings (OpenVibe.AI reads them through signed URLs) |
| 44 | |
| 45 | ## Depends on |
| 46 | |
| 47 | - OpenVibe.Network (JWKS, visitor SSO as OAuth client `media`, `identity.subject.resolve`, account |
| 48 | export and deletion), OpenVibe.Live (`live.lineage.resolve`), OpenVibe.Events (the outbox relay and |
| 49 | the account and revocation subscriptions) |
| 50 | - Backblaze B2 and Cloudflare R2 (S3 API) when configured; `ffmpeg`/`ffprobe` on the host |
| 51 | - `openvibe-contracts` v0.423.0, `openvibe-sdk` v0.161.0 (tokens, events outbox, usage readings, per-actor limits), |
| 52 | `openvibe-shared` v4.241.0, pinned by release tarball |
| 53 | |
| 54 | ## Capabilities |
| 55 | |
| 56 | Implemented here (the service manifest's `capabilities`, audience `openvibe.media`, checked per |
| 57 | namespace against the token's `ns`): `media.object.upload`, `media.object.read`, `media.object.list`, |
| 58 | `media.object.delete`, `media.upload.create`, `media.derivative.create`, `media.derivative.read`, |
| 59 | `media.lifecycle.read` and `media.lifecycle.transition`. First-party apps also use their tenant API |
| 60 | key ([Tenancy & auth](#tenancy--auth)). |
| 61 | |
| 62 | | Capability | Audience | Route | Notes | |
| 63 | |---|---|---|---| |
| 64 | | `media.resource.read` | first-party | `GET /api/v1/resources`, `GET /api/v1/resources/:ovrn` | the authority resource index ([Resource index](#resource-index-apiv1resources-first-party)). Active in openvibe-contracts since 0.108.0; the route enforces it (a Network service token holding it, loopback, audience `openvibe.media`) | |
| 65 | | `media.resource.control` | first-party | `POST /api/v1/resources/control` | `delete` a person's (or their project's) object: the soft delete, restorable until its retention ends ([Resource index](#resource-index-apiv1resources-first-party)). Active in openvibe-contracts since 0.169.0 | |
| 66 | | `media.object.s3` | public (an access key) | the S3 surface at `s3.openvibe.media` ([docs/s3.md](docs/s3.md)) | PutObject, GetObject, HeadObject, DeleteObject(s), ListObjects(V2), ListBuckets, HeadBucket, presigned GET/PUT; a bucket is a zone | |
| 67 | | `media.access_key.manage` | first-party | `POST/GET/DELETE /api/v1/projects/:project/access-keys` | S3 access keys for a project admin, the secret shown once (Zone's console, Services) | |
| 68 | | `media.zone.provision` | first-party (Zone) | `PUT/GET /internal/zones/:id` | OpenVibe.Zone's zone records, applied by revision; usage and deletion progress | |
| 69 | |
| 70 | Called elsewhere, as the service principal `media` (the OAuth client `media`): |
| 71 | |
| 72 | | Service | Grant | Why | |
| 73 | |---|---|---| |
| 74 | | OpenVibe.Network | `identity.subject.resolve`; `network.account.export.contribute`, `network.account.deletion.confirm` | owner subjects for objects; account export and deletion (ADR-033) | |
| 75 | | OpenVibe.Live | `live.lineage.resolve` | which channel and person a recording belongs to | |
| 76 | | OpenVibe.Events | `events.event.publish`, `events.subscription.manage` | the outbox relay; its subscriptions | |
| 77 | |
| 78 | Media announces outcomes as OpenVibe.Events events (`media.vod.ready`, …) and sends no webhooks since 2026-10-11 (C-75). |
| 79 | |
| 80 | ## Layout |
| 81 | |
| 82 | ``` |
| 83 | server/ |
| 84 | index.js boot, route mounts, background jobs, graceful shutdown |
| 85 | config.js env config (PORT, MEDIA_DATA_DIR, *_PATH, DATABASE_URL/DATABASE_DIRECT_URL, MEDIA_B2_*/R2_*, |
| 86 | RTP pool, OV_NETWORK_URL) |
| 87 | auth.js tenancy + auth middleware, JWKS fetch, app seeding |
| 88 | announce.js transactional OpenVibe.Events outcome announcements |
| 89 | drill.js MEDIA_DRILL: restore-drill mode (reads only, no bytes, no jobs, nothing leaves) |
| 90 | db/database.js PostgreSQL helpers (openvibe-sdk/db, async), all app_id-scoped |
| 91 | db/init.js applies migrations/ (DATABASE_URL/DATABASE_DIRECT_URL, or embedded PGlite) |
| 92 | migrations/NNNN_*.sql apps, vods, clips, pastes(+likes/comments), files, content_views, media_settings, |
| 93 | media_objects/locations/relationships/variants/jobs/holds/invariant_violations |
| 94 | vod/recorder.js ffmpeg recording: RTMP pull + RTP (SDP) ingest, codec passthrough |
| 95 | vod/media-tools.js probes, seekable remux, DVR sidecar, chunk-segment concat |
| 96 | vod/finalize.js finalize pipeline (remux → probe → master recovery → thumbnail → event) |
| 97 | vod/routes.js /api/v1/:app/vods (+ ingest + chunks) |
| 98 | vod/clip-cutter.js re-encoding clip cutter (local file or presigned URL source) |
| 99 | vod/clips-routes.js /api/v1/:app/clips |
| 100 | vod/vod-storage.js local/B2/R2 tiering, presigned playback, sweep, CLI |
| 101 | vod/health-scanner.js probe/decode/master-recovery primitives |
| 102 | vod/health-job.js background health scan + quarantine cleanup + master sweep |
| 103 | files/routes.js /api/v1/:app/files |
| 104 | admin/routes.js /api/v1/:app/admin/storage (disk, tiers, buckets, bulk ops) |
| 105 | registry/resource-index.js the authority resource index (/api/v1/resources; common.resource-summary@1, ADR-048) |
| 106 | registry/resource-control.js the control API (POST /api/v1/resources/control; delete an object, ADR-048) |
| 107 | registry/projects.js whether a person administers a project (Network's internal membership read) |
| 108 | thumbnails/ thumbnail service + /api/v1/:app/thumbnails |
| 109 | public/routes.js public /v /c /p /t /f |
| 110 | me/ the object explorer: /me and /api/v2/me (a signed-in person's own objects, uploads, |
| 111 | derivatives and usage) and the operator views /me/ops — see docs/object-model.md#object-explorer |
| 112 | objects/ canonical object model: model (projections, holds), routes (/api/v2 + /o), |
| 113 | reconcile, invariant, signing (presigned URLs), multipart, |
| 114 | content-type, verify-job (scheduled copy verification), copy-report, |
| 115 | readiness (metadata / bytes_verified / playable), popularity (unique viewers |
| 116 | per day, no identifiers), tier-policy + tiering (native objects in the R2 cache) — |
| 117 | see docs/object-model.md |
| 118 | jobs/ media_jobs: queue (states, idempotency, media.job.* events in the state |
| 119 | change's transaction; payload queue.jobEvent: ids, state, ISO times, |
| 120 | has_result, never params/result/error text/creator), worker (light/heavy lanes, leases, retries, |
| 121 | cancellation), routes (/api/v2/:app/jobs), types: thumbnail.regenerate, |
| 122 | invariant.scan (proposes split/remux), object.split, object.remux, object.waveform, object.sprite, |
| 123 | object.cmaf (CMAF/HLS segments of a finished video into media_timeline; MEDIA_HLS_ENABLED), |
| 124 | object.pack (durable segments packed into ~60 s chunk objects; MEDIA_HLS_ENABLED), |
| 125 | rendition.create (an on-demand 720p H.264/AAC rendition of a media object into media_timeline; |
| 126 | MEDIA_HLS_ENABLED + MEDIA_RENDITIONS), |
| 127 | clip.cut (a clip's cut/re-cut; with MEDIA_HLS_ENABLED + MEDIA_MATERIALIZED_CLIPS and a source |
| 128 | timeline it materializes over the source's segments, only the edges re-encoded — |
| 129 | docs/materialized-clips.md), |
| 130 | vod.finalize (orphans and failed finalizes, with backoff), |
| 131 | vod.duration.reconcile (stored vs measured durations, local and B2/R2), |
| 132 | storage.orphans.scan (monthly storage orphan report, service-wide, report only), |
| 133 | storage.move.cleanup (the alert for placement moves whose delete keeps failing, service-wide) |
| 134 | client-ip.js trust proxy = loopback; req.ip is the only client address |
| 135 | (openvibe-shared v4.241.0, openvibe-contracts v0.419.0, openvibe-sdk v0.161.0: pinned release tarballs, installed by npm) |
| 136 | scripts/smoke-test.sh end-to-end smoke test (boots a temp instance) |
| 137 | scripts/reconcile-objects.js / object-invariant.js / no-good-copy-report.js object-model operator tools |
| 138 | scripts/media-jobs.js list jobs, run the size-invariant scan (dry run by default), approve/cancel proposals |
| 139 | scripts/r2-eviction-drill.js evict one VOD's R2 copy, prove B2 serves it, re-warm R2; JSON artifact (dry run by default) |
| 140 | scripts/vod-duration-reconcile.js stored VOD durations vs the real files (dry run by default; --apply --backup; --rollback) |
| 141 | scripts/vods-orphans-report.js read-only report on the B2 vods-orphans/ prefix (matches, sizes, recommendations); |
| 142 | --storage: the whole storage against the database (files and keys no row names, |
| 143 | copies that are not there, open multipart uploads), as the monthly job does |
| 144 | ``` |
| 145 | |
| 146 | ## Visitor sign-in |
| 147 | |
| 148 | The public pages use the shared navbar and footer. Visitors sign in through the Network as OAuth |
| 149 | client `media` (`server/user-auth.js`, routes under `/auth`, host-only cookies); set |
| 150 | `OV_OAUTH_CLIENT_SECRET` in the unit's environment. `/terms`, `/privacy` and `/dmca` come from |
| 151 | `openvibe-shared/legal` (profile `hosting`). This is separate from tenant auth below, which is how |
| 152 | other OpenVibe services call the API. |
| 153 | |
| 154 | **Your media (`/me`).** A signed-in person sees the objects whose `owner_subject` is their Network |
| 155 | subject, in every tenant: uploads in progress, copies, derivatives, visibility, lifecycle and their own |
| 156 | usage per tenant and namespace (`GET /api/v2/me/objects`, `/objects/:id`, `/usage`). Read-only: changes |
| 157 | stay with the apps. Network staff with `staff.site.view` get the operator views at `/me/ops` (failed |
| 158 | jobs, missing media, projection, tiering and usage recompute with `staff.site.configure`). Details in |
| 159 | [docs/object-model.md](docs/object-model.md#object-explorer). |
| 160 | |
| 161 | ## Tenancy & auth |
| 162 | |
| 163 | Tenant = the `:app` path segment. Rows in the `apps` table define tenants: |
| 164 | |
| 165 | | column | meaning | |
| 166 | |---|---| |
| 167 | | `app_id` | tenant id (`live`, `games`, …) | |
| 168 | | `api_key_hash` | sha256 of the app API key (never stored in plaintext) | |
| 169 | | `webhook_url` / `webhook_secret` | retained for a separate schema migration; unused for delivery | |
| 170 | | `allowed_origins` | JSON array; guards browser (user-JWT) calls | |
| 171 | | `quota_bytes` | storage quota: v1 files + native v2 objects (0 = unlimited) | |
| 172 | | `project_id` / `env` | set only on developer-project tenants (below) | |
| 173 | |
| 174 | Credential types on `/api/v1/:app/...`: |
| 175 | |
| 176 | 1. **App API key** — `Authorization: Bearer <app_api_key>`. Compared in |
| 177 | constant time against the app's stored hash. A key is only valid for its |
| 178 | own `:app` segment; presenting another app's key returns **403**. |
| 179 | 2. **Network user JWT** (browser endpoints only — chunks/complete, clips, |
| 180 | files, thumbnails) — RS256, verified **offline** against the JWKS |
| 181 | public key fetched from `OV_NETWORK_URL/api/.well-known/jwks` at boot |
| 182 | (cached, refreshed every 6 h). `aud` must include `openvibe.media` when |
| 183 | present. If the request carries an `Origin` header it must be in the app's |
| 184 | `allowed_origins` (CORS is reflected for allow-listed origins only). |
| 185 | |
| 186 | 3. **Network principal token** (`sub svc:…`, `app:…`, `mod:…`; RS256, audience `openvibe.media`) — |
| 187 | accepted only on routes that name a verb, each checked for the namespace it works in against the |
| 188 | token's `ns`: read (`media.object.read`), list (`media.object.list`, or read), write |
| 189 | (`media.object.upload`), delete (`media.object.delete`, or upload) and transform |
| 190 | (`media.derivative.create`, or upload). The tenant's root namespace is `:app`; children, quotas |
| 191 | and policy are in [docs/object-model.md](docs/object-model.md#namespaces-grants-and-quotas). |
| 192 | |
| 193 | ### Developer-project tenants (ADR-014) |
| 194 | |
| 195 | A developer app from OpenVibe.Network (token `sub app:app_<ULID>`, `project_id prj_<ULID>`, |
| 196 | `env sandbox|production`, `ns [project_id, app.<project_id>.*]`; an older `ns [project_id]` means the |
| 197 | same) holding media verbs gets a tenant keyed by its project id, **created on first use**. Its |
| 198 | namespaces are `app.<project_id>` (production) and `app.<project_id>.sandbox`, with children below. The URL always names the project |
| 199 | (`/api/v1/prj_<ULID>/files`, `/api/v2/prj_<ULID>/objects`, so openvibe-sdk's |
| 200 | `createMediaClient({ app: projectId })` works); the token's `env` picks one of two separate tenants: |
| 201 | |
| 202 | | env | tenant (`apps.app_id`) | default quota | |
| 203 | |---|---|---| |
| 204 | | production | `prj_<ULID>` | `MEDIA_APP_TENANT_QUOTA_MB` (1024) | |
| 205 | | sandbox | `prj_<ULID>-sandbox` | `MEDIA_APP_SANDBOX_QUOTA_MB` (100) | |
| 206 | |
| 207 | - Both rows carry `project_id` and `env`, and have no API key (`api_key_hash` empty; the seeding |
| 208 | env refuses `prj_…` ids). Only the project's own app tokens reach them: other projects' tokens, |
| 209 | first-party service tokens, app keys and user JWTs get 403/404, and the `-sandbox` id cannot be |
| 210 | named in a URL. Nothing is created unless the token's `project_id` is the path's project and the |
| 211 | token holds the route's capability. |
| 212 | - App tokens reach only these tenants and only the capability routes (files, objects v2 without |
| 213 | retention holds). VODs, clips, thumbnails, assets, stats and admin refuse them. |
| 214 | - **Sandbox tokens** (`env: sandbox`) are accepted on these routes only; everywhere else they get |
| 215 | `401 token.sandbox_refused`. |
| 216 | - **Sandbox content is never public.** `/f/:key` and `/o/:id` answer 404 for it unless the URL |
| 217 | carries a valid signature; the Files API returns a signed, short-lived `url` (+ `url_expires_at`, |
| 218 | `sandbox: true`), objects have `public_url: null` and `/download` always signs, whatever the |
| 219 | visibility. Sandbox tenants emit no platform events. |
| 220 | - The quota counts v1 files and native v2 objects together (files uploads now use the same count |
| 221 | as objects for every tenant). v1 file keys in project tenants carry a tenant tag, so two tenants |
| 222 | never collide on identical uploads. Project tenants' files are not listed in the public gallery. |
| 223 | Soft-deleted objects keep counting against a project tenant's quota until their bytes are purged. |
| 224 | - `/f/:key` (like `/o/:id`) sends `nosniff` and serves only images (not SVG), video, audio, PDF and |
| 225 | plain text inline; anything else an uploader labels (HTML, SVG, XML…) is an `attachment`. |
| 226 | - Not yet: reading quotas set in Network (`network.project.read`), and removing a project's |
| 227 | tenants when the project is archived. |
| 228 | |
| 229 | ### First-party tenants |
| 230 | |
| 231 | Boot creates any missing first-party tenant: `live`, `community`, `tools`, `games`, `wiki`, `blog`, `inventory` and |
| 232 | `media-hub` (`server/auth.js` `ensureTokenOnlyApps`), with its quota and browser origins, and leaves an existing row as |
| 233 | it is. Each is reached only with its service's Network service token, with grants per namespace in Network. No app |
| 234 | key is issued or seeded: `MEDIA_APPS_SEED` and `MEDIA_APP_KEYS` are gone (compatibility register C-71), and migration |
| 235 | `0006` emptied the key hashes the tenants still carried. App keys now exist only as test fixtures (`db.upsertApp`) |
| 236 | and in `scripts/smoke-test.sh`. |
| 237 | |
| 238 | ## API (all under `/api/v1/:app`) |
| 239 | |
| 240 | ### VODs |
| 241 | |
| 242 | | method | path | notes | |
| 243 | |---|---|---| |
| 244 | | POST | `/vods` | `{ title?, stream_id?, stream_key?, managed_stream_id?, user_id?, meta?, visibility?, clips_only? }` → `{ id }` | |
| 245 | | POST | `/vods/:id/ingest/rtmp` | `{ rtmp_url }` → **202**; ffmpeg pulls the URL, lossless stream-copy → fragmented `.mp4`. Only allow-listed RTMP servers (`MEDIA_RTMP_PULL_ALLOW`, a list of `host:port` or `host:low-high`, default `127.0.0.1:1935,localhost:1935,[::1]:1935,127.0.0.1:19360-19399`: Live's RTMP server and OpenRestream's loopback play ports on this host) and a plain `rtmp[s]://host[:port]/path` shape (no user info, whitespace or backslash); anything else is **400** and starts nothing | |
| 246 | | POST | `/vods/:id/ingest/rtp/start` | `{ video: {payloadType, codec, clockRate, ssrc?, parameters?}, audio?: {…} }` → `{ videoPort, audioPort }` from UDP 12000-12199; point PlainRtpTransports at `127.0.0.1` (RTCP = port+1) | |
| 247 | | POST | `/vods/:id/ingest/rtp/stop` | finalizes the recording | |
| 248 | | POST | `/vods/:id/chunks` | multipart `chunk` (+`segmentId`), user JWT ok — browser MediaRecorder append flow | |
| 249 | | POST | `/vods/:id/complete` | finalize chunked upload (user JWT ok) | |
| 250 | | POST | `/vods/:id/finalize` | close recording; remux, probe, thumbnail, event | |
| 251 | | GET | `/vods/:id` | `{ id, watch_id, title, status, duration, playback_url, hls_url, thumbnail_url, storage_provider, readiness, … }` (`watch_id`: the item's 11-character id on openvibe.video's `/watch?v=` page, [below](#watch-ids); `readiness`: [below](#readiness); `hls_url`, with `hls_expires_at` when signed: the object's master playlist with `MEDIA_HLS_ENABLED`, docs/media-fabric.md §3) | |
| 252 | | GET | `/vods?limit&offset&user_id&stream_id&managed_stream_id&include_private&order&since` | list; `include_private` app-key only; `order` = newest\|oldest\|views; `since` = created at or after (ISO 8601 or `YYYY-MM-DD HH:MM:SS`, UTC) | |
| 253 | | PUT | `/vods/:id` | `{ title?, description?, visibility? }` | |
| 254 | | DELETE | `/vods/:id` | deletes local + B2 + R2 objects + row | |
| 255 | |
| 256 | `status`: `pending → recording → ready | failed` (derived; failures come from |
| 257 | health quarantine: corrupt / zero-byte / missing file). |
| 258 | |
| 259 | <a id="watch-ids"></a>**Watch ids.** Every VOD and clip also answers to an 11-character watch id: |
| 260 | 8 random bytes as URL-safe base64 (`crypto.randomBytes(8).toString('base64url')`), YouTube's shape, never |
| 261 | derived from the numeric id. It is what openvibe.video's `/watch?v=<watch_id>` page is addressed by, and it |
| 262 | exists so a watch page cannot be found by counting. Media owns VODs and clips, so Media mints the id |
| 263 | (`server/watch-ids.js`): at insert for every new row, and in idempotent batches for rows that predate it (a |
| 264 | boot backfill, only where `watch_id IS NULL`). One id space across both tables: `mintUnique()` checks both |
| 265 | before it answers, so a watch id has exactly one meaning. |
| 266 | |
| 267 | `GET /api/v1/:app/watch/:watch_id` → `{ kind: 'vod' | 'clip', id: <numeric id> }` (404 when no row holds |
| 268 | it). Visibility is unchanged: a private item still resolves here, and the item's own read applies the rule it |
| 269 | always did. An unlisted item stays reachable only by its watch id, never listed. |
| 270 | |
| 271 | <a id="readiness"></a>**Readiness.** VOD and clip answers (and v2 objects) also carry `readiness` |
| 272 | `{ metadata, bytes_verified, playable, hash_verified, verified_copies, reason }`, read from the object |
| 273 | model: `bytes_verified` needs a copy whose bytes a check found (local file, B2/R2 HEAD, sha256 when |
| 274 | known), and `playable` adds a finished, not failed, not deleted VOD or clip. A B2 copy nobody has |
| 275 | checked yet is not verified. The watch pages offer a player only when `playable` (see |
| 276 | [docs/object-model.md](docs/object-model.md#readiness)). |
| 277 | |
| 278 | **Durations are measured, never estimated.** Finalize stores what the file says and |
| 279 | records where it came from in `duration_source`: `probe` (ffprobe's container |
| 280 | duration), `remux` (the last packet time of a stream-copy pass, used when the header |
| 281 | has none or is far longer than the packets) or `unknown` (nothing measurable: the |
| 282 | duration is stored as **0**, `health_status` is `needs_review` with `probe_failed` or |
| 283 | `inflated_duration`, and the VOD is hidden until a later finalize measures it). The |
| 284 | wall clock only bounds a recording the recorder saw start (a value longer than 1.5x |
| 285 | its run + 30 s is refused); an orphan finalized after a restart has no bound and is |
| 286 | never measured as now - created_at. While recording, `duration` is the live elapsed |
| 287 | time; finalize replaces it. |
| 288 | |
| 289 | **Recording formats** (inherited codec-passthrough behavior — VODs are *not* |
| 290 | always `.webm`): RTMP and RTP-H.264 record by lossless stream copy into a |
| 291 | fragmented **`.mp4`** (audio → AAC on the RTP path); RTP-VP8/VP9 copies into |
| 292 | **`.webm`** (Opus copied); exotic codecs fall back to a libvpx re-encode with a |
| 293 | lossless `.master.mkv` recovery archive. A `.seekable` sidecar is remuxed every |
| 294 | 60 s during recording so `/v/:id` is DVR-seekable while live. |
| 295 | |
| 296 | ### Clips |
| 297 | |
| 298 | | method | path | notes | |
| 299 | |---|---|---| |
| 300 | | POST | `/clips` | `{ vod_id, start_s, end_s, title?, description?, user_id?, visibility?, auto_generated?, materialize? }` → over a VOD with a CMAF timeline (`MEDIA_HLS_ENABLED`): a **virtual** clip, **201** ready at once (the clip, `storage_provider: 'timeline'`), played as HLS over the source's segments (`/c/:id` redirects to its playlist); with `materialize: true` or no timeline → **202** `{ id, status: 'processing', job_id }`; cut (`clip.cut`) runs in background — with `MEDIA_HLS_ENABLED` + `MEDIA_MATERIALIZED_CLIPS=1` and a source timeline it is **materialized** over the source's bytes (the clip's own media_timeline rows copy the interior segments; only the two window edges are re-encoded, docs/materialized-clips.md), otherwise the full re-encode (from the local file or a presigned B2/R2 URL); duplicate windows are deduplicated; live recordings are clamped to flushed footage. Multipart `video` = direct upload of an already-cut blob → **201** ready | |
| 301 | | GET | `/clips/:id` | status: `processing | ready | failed`; with `MEDIA_HLS_ENABLED` also `hls_url` (and `hls_expires_at` when signed) — the clip's playlist over its own timeline (a materialized clip's persisted rows) or, virtual, its source's window (docs/media-fabric.md §4) | |
| 302 | | POST | `/clips/:id/recut` | `{ materialize? }` → a file-less clip over a timeline turns virtual (200, the clip); otherwise (or `materialize: true`) **202** `{ id, status, job_id }` (`clip.cut`; a virtual clip stays ready while it is cut) | |
| 303 | | POST | `/clips/:id/trim` | `{ start_s, end_s }`, seconds into the clip (at least 1 s apart) → the clip narrows in place: same id and title, its window of the source shrinks and it is cut again from the source (**202** `{ id, status: 'processing', job_id }`, `clip.cut`, `clip.ready` again; the superseded file is removed), or a virtual clip plays the narrower window at once (200, the clip); the whole clip → 200 `unchanged: true`; a file clip is `processing` from the answer until its cut is in; still being cut (or a queued/running `clip.cut`) → 409; under a retention hold (its own or its VOD's) → 409 `media.object.held`; no source VOD (a browser upload), a file moved to cloud storage or an unreadable source recording → 422; a cut that fails for good gives the clip its previous window and file back (`clip.ready` again) | |
| 304 | | GET | `/clips?limit&offset&vod_id&stream_id&user_id&channel_user_id&hide_self&include_private&order&auto_generated&status&since` | list; `channel_user_id` = clipped-channel owner; `include_private` app-key only; `auto_generated=1\|0` = cut by the app's automation (AI auto-clips) or made by a person; `status=ready` = playable clips only; `since` as for VODs | |
| 305 | | PUT | `/clips/:id` | `{ title?, visibility?, auto_generated? }` (`auto_generated` app key only, 403 when acting for a user) | |
| 306 | | DELETE | `/clips/:id` | local + offloaded objects + row; a timeline clip's own rows and edge bytes go with it, while every location the source still names — a materialized clip's interior — stays (docs/materialized-clips.md) | |
| 307 | |
| 308 | ### Pastes |
| 309 | |
| 310 | **The API is gone:** OpenVibe.Community owns pastes (since 2026-09-22) and Media's |
| 311 | read-only paste app API was retired in T10 step 2 — `/api/v1/:app/pastes` is |
| 312 | unmounted, so every method there answers 404. No paste is created, read, listed or |
| 313 | deleted through Media any more. |
| 314 | `/p/:slug` and `/p/:slug/raw` are unconditional **301s** to Community: the target |
| 315 | is `config.pastes.movedTo` (`PASTES_MOVED_TO`, default `https://openvibe.community`, |
| 316 | so the redirect never depends on an env var) and no row is read — a private or |
| 317 | missing slug redirects exactly like a public one. Legacy screenshot and avatar objects are |
| 318 | found by `legacy_ref` and `/p/:slug/screenshot` redirects to `/o/<id>` when they are |
| 319 | public or unlisted and ready. Other slugs redirect to Community. Avatar ingest stores |
| 320 | new avatars as native objects in Live's avatars namespace. Legacy files remain under |
| 321 | `PASTES_PATH` and are named by their objects' local locations. Legacy screenshot and |
| 322 | avatar objects are deleted through account deletion; direct v2 deletion answers 409. |
| 323 | |
| 324 | | method | path | notes | |
| 325 | |---|---|---| |
| 326 | | GET | `/p/:slug` | always 301 to OpenVibe.Community — pastes live there; no row is read | |
| 327 | | GET | `/p/:slug/raw` | always 301 to Community's `/p/:slug/raw`; the suffix is preserved | |
| 328 | | GET | `/p/:slug/screenshot` | 301 to the legacy object when public or unlisted and ready; otherwise 301 to Community | |
| 329 | |
| 330 | Legacy paste tables remain in the database until the later contract migration, but Media does not query them. |
| 331 | |
| 332 | ### Files |
| 333 | |
| 334 | | method | path | notes | |
| 335 | |---|---|---| |
| 336 | | POST | `/files` | multipart `file` → `{ key, url, size, mime }`; key = `<sha256[0:12]>-<name>`; per-app `quota_bytes` enforced (413 on exceed) | |
| 337 | | GET | `/files` | list + `used_bytes`/`quota_bytes` | |
| 338 | | GET | `/files/:key` | meta | |
| 339 | | DELETE | `/files/:key` | delete | |
| 340 | |
| 341 | ### Thumbnails |
| 342 | |
| 343 | | method | path | notes | |
| 344 | |---|---|---| |
| 345 | | POST | `/thumbnails/:kind/:id` | kind `vod`/`clip`: multipart `thumbnail` (or `{image: base64}`) uploads a custom image, or with no body (re)generates from the media (VOD @10%, clip near first frame); kind `live`: upload a broadcaster frame, stored under the stable name `stream-<app>-<id>.jpg` → `{ url }` | |
| 346 | |
| 347 | ### Admin storage (`/admin/storage`, app-key auth only) |
| 348 | |
| 349 | Storage-management endpoints ported from the predecessor's admin panel. Auth |
| 350 | is the app API key only (no user JWTs) — admins reach these through their |
| 351 | app's own server, which holds the key. DB-derived stats are scoped to the |
| 352 | calling app; **disk totals and directory sizes are host-wide** (the data |
| 353 | directories are shared across apps) and responses carry a `note` saying so. |
| 354 | |
| 355 | | method | path | notes | |
| 356 | |---|---|---| |
| 357 | | GET | `/admin/storage` | `{ disk (df of VOD volume), database.bytes, breakdown [vods/clips/pastes/thumbnails/files dirs], vodStats/clipStats/fileStats (app-scoped), byProvider.{vods,clips} (count+bytes per local/b2/r2) }` | |
| 358 | | GET | `/admin/storage/vods?limit&offset&sort&order&provider&tier` | detailed app VOD listing: id, title, size, provider, health, views, `last_accessed_at`, created + `fileExists`/`diskSize`/`actualTier` (disk-reconciled: local/b2/r2/missing), per-user summary. `sort` = size\|date\|duration\|tier\|views\|accessed, `order` = asc\|desc, `provider` = local\|b2\|r2 (`tier` accepts legacy hot/cold aliases) | |
| 359 | | DELETE | `/admin/storage/vods/bulk` | `{ ids: [...] }` (max 200) — deletes each VOD everywhere (local + B2 + R2 + sidecars/master + thumbnail + row), app-scoped → `{ deleted, freed, results: [{id, ok, error?}] }` | |
| 360 | | GET | `/admin/storage/tiers` | tiering status: settings, provider health, local disk, service-wide tier counts (`tiers`/`clipTiers`), `sweepRunning`, plus `app.{tiers, pendingOffload}` scoped to the caller | |
| 361 | | PUT | `/admin/storage/tiers/settings` | update any `storage_tier.*` knob (see `vod-storage.js` `DEFAULTS`: `enabled`, `minAgeDays`, `maxViewsForCold`, `minLastAccessDays`, `sweepIntervalMs`, `hotDiskPressurePct`, `localLowWaterPct`, `maxPerSweep`, `r2Enabled`, `r2MinViews`, …); persisted in `media_settings`, sweep timer restarted → `{ ok, settings }` | |
| 362 | | POST | `/admin/storage/tiers/sweep` | run the tiering sweep now → sweep summary | |
| 363 | | POST | `/admin/storage/tiers/move` | `{ vod_id, target: local\|hot\|b2\|cold\|r2 }` — reuses the storage engine's verified move logic; VOD must belong to the app | |
| 364 | | POST | `/admin/storage/tiers/bulk-move` | `{ ids: [...], target }` (max 50) → `{ moved, bytes, errors? }` | |
| 365 | | GET | `/admin/storage/tiers/policy` | **read-only R2 policy**: each threshold (`r2Enabled`, `r2MinViews`, `r2RecentAccessDays`, `r2MaxIdleDays`, `r2MaxPerSweep`) as `{ value, default, source: default\|setting }`, the promote and demote rules in words, the R2/B2 provider status, and this app's decisions: `last_24h` counts per action and outcome and the 20 most recent | |
| 366 | | GET | `/admin/storage/tiers/decisions?vod_id&action&outcome&limit&before_id` | this app's R2 promote/demote decisions, newest first: `{ id, decided_at, vod_id, object_id, action, from_provider, to_provider, outcome: done\|already\|refused\|failed, trigger: sweep\|admin\|clip\|drill\|manual, reason, inputs, thresholds, error }` → `{ decisions, next_before_id }` | |
| 367 | | GET | `/admin/storage/tiers/objects/policy` | native v2 objects' R2 tiering (below): the activation gate, each `media.object_tier` threshold as `{ value, default, source }`, the rules, this app's promotion candidates (with what would refuse each) and its decisions (24-hour counts and the 20 most recent) | |
| 368 | | GET | `/admin/storage/tiers/objects/decisions?object_id&action&outcome&limit&before_id` | this app's native object decisions, newest first (`outcome`: done\|already\|refused\|failed\|dry_run) → `{ decisions, next_before_id }` | |
| 369 | | GET / POST | `/admin/storage/config[/media.storage_tier\|/media.object_tier[/history\|/rollback]]` | the revisioned tier policies (openvibe-shared/config): read, apply `{ values, merge, reason }`, history, roll back | |
| 370 | | GET | `/admin/storage/holds?all&object_id&vod_id&clip_id&limit&before_id` | this app's retention holds, newest first (active only unless `all=1`) | |
| 371 | | POST | `/admin/storage/holds` | `{ object_id \| vod_id \| clip_id, reason, kind?, note?, placed_by? }` → 201: the object is kept (no delete, no tier move) and so are the clips cut from a held VOD. Not for calls acting for a user (403) | |
| 372 | | POST | `/admin/storage/holds/:holdId/release` | `{ released_by }` (also `DELETE /admin/storage/holds/:holdId`); 409 when already released. Holds stay on record | |
| 373 | | GET | `/admin/storage/buckets` | sanitized bucket status per provider: `{ configured, endpoint, bucket, region, healthy, reachable }` via a live HeadBucket probe — **credentials are never returned** | |
| 374 | | GET | `/admin/storage/ops` | this app's operator report (as `/me/ops` shows it to Network staff): failed jobs, missing media, projection, tiering, the namespaces' usage snapshot. Not for calls acting for a user (403) | |
| 375 | | POST | `/admin/storage/ops/recompute` | refresh this app's namespaces' usage snapshot from the rows | |
| 376 | |
| 377 | ### Outcome events |
| 378 | |
| 379 | Media announces outcomes as OpenVibe.Events events (`media.vod.ready`, …) and sends no webhooks since 2026-10-11 (C-75). |
| 380 | |
| 381 | Each outcome is a durable OpenVibe.Events event (`media.vod.ready|failed`, |
| 382 | `media.clip.ready|failed`, `media.object.uploaded`, `media.storage.alert|recovered`; |
| 383 | `server/events.js`). The outbox row is written in the **same transaction** as the state |
| 384 | change it describes (`announce()`), so an outcome is never lost or announced for a |
| 385 | rolled-back change. |
| 386 | |
| 387 | Two additional object events go to OpenVibe.Events: **`media.object.deleted`** |
| 388 | `{ object_id, app_id, kind, legacy_ref, deleted_at }` and **`media.object.visibility_changed`** |
| 389 | `{ object_id, app_id, kind, legacy_ref, visibility, previous_visibility, changed_at }` (subject |
| 390 | `object <id>`, `important`, times ISO 8601 UTC, `legacy_ref` null for native objects). Nothing |
| 391 | else about the object is in them (no title, owner, keys or sizes). Triggers on `media_objects` |
| 392 | stage every deletion and visibility change in the transaction that makes it, whichever path made it |
| 393 | (a v1 visibility change, a row delete, a v2 soft delete, operator SQL); the staged rows become outbox |
| 394 | envelopes inside Media's own transactions (projection sync, `announce()`, soft delete), so those |
| 395 | commit with their change, and the relay drains anything else. No visibility event for an object |
| 396 | that is deleted; none for sandbox tenants. `test/object-events.test.js`. |
| 397 | |
| 398 | ## Resource index (`/api/v1/resources`, first-party) |
| 399 | |
| 400 | Media's authority resource index (ADR-048 §3, plan T13 step 8): the resources Media owns, as |
| 401 | `common.resource-summary@1` — the shape OpenVibe.Services fans out over and merges. It lists Media's |
| 402 | **objects alone** (`kind` `media.object`, id `med_<ULID>`, the `media_objects` table): v1 vods, clips and |
| 403 | files are projections over objects (bigint ids) and are never listed. |
| 404 | |
| 405 | ``` |
| 406 | GET /api/v1/resources[?project=<prj_ id>&kind=<kind>&cursor=<opaque>&limit=<n>] → common.resource-list-result@1 |
| 407 | GET /api/v1/resources/:ovrn → common.resource-summary@1 |
| 408 | ``` |
| 409 | |
| 410 | | Param | Meaning | |
| 411 | |---|---| |
| 412 | | `project` | the tenancy boundary: a `prj_` id. Only that project's rows answer (its production and sandbox tenants); never another project's. Omitted, the first-party caller sees every Media resource | |
| 413 | | `kind` | `media.object` (the only kind Media owns). An unknown kind is an empty page, not an error | |
| 414 | | `cursor` | opaque keyset cursor from the previous page's `next_cursor`, ordered by id | |
| 415 | | `limit` | page size, default 100, max 1000 | |
| 416 | |
| 417 | Auth is a Network service token holding `media.resource.read`, loopback only (audience `openvibe.media`; |
| 418 | `server/service-guard.js`) — the same first-party check as the other internal routes. No token is |
| 419 | 401, a token without the capability 403. A bad query is 400 `resources.bad_query`; an unknown or |
| 420 | unnameable resource is 404 `resources.unknown_resource`. |
| 421 | |
| 422 | An object's `project_id` is its tenant's (`apps.project_id`); a first-party tenant (`live`, …) has none, so |
| 423 | its objects carry no `project_id` and no `ovrn`. The summary's `ovrn` is composed by openvibe-contracts' |
| 424 | `contracts.resources.nameOf` — the one formatter — so it is present exactly when the object has a project |
| 425 | (`ovrn:media:<project_id>:object/<med_…>`); `GET /api/v1/resources/:ovrn` reads one by that name, and only a |
| 426 | resource whose computed name equals the one asked for answers. `owner` is a `subject-ref` when the object |
| 427 | stores a `usr_` owner. `test/resource-index.test.js`. |
| 428 | |
| 429 | **Control** (`media.resource.control`, plan T13 step 10): `POST /api/v1/resources/control` takes a |
| 430 | `common.resource-control-request@1` from OpenVibe.Services (same loopback service-token check, and the caller must |
| 431 | be a `svc:` service) and answers a `common.resource-control-result@1`. The one action is `delete`: the API's soft |
| 432 | delete (`restorable_until` in the result; restore through `POST /api/v1/objects/:id/restore`). A held object and a |
| 433 | legacy-managed one (a VOD, clip or file) are refused. The object's owner may act, and so may an admin or owner of its |
| 434 | project (asked of Network with `network.project.member.read`, `server/registry/projects.js`); anyone else gets |
| 435 | `resources.unknown_resource`. Dry runs; the first answer per (caller, project, key) is kept in `media_control_log` and |
| 436 | returned for a repeat. `test/resource-control.test.js`. |
| 437 | |
| 438 | ## Restore drills (`MEDIA_DRILL=1`) |
| 439 | |
| 440 | A restore drill starts a second Media from this checkout on 127.0.0.1:14100 with `MEDIA_DRILL=1` and |
| 441 | `DATABASE_URL` naming a restored copy of the database (pgBackRest restores the cluster on a spare port). In that |
| 442 | mode (`server/drill.js`) Media: |
| 443 | |
| 444 | - refuses to start unless `DATABASE_URL` is set and does not name production's database (`ov_media` on 5432, or |
| 445 | PgBouncer's 6432), `DATABASE_DIRECT_URL` is unset (a drill never migrates), `HOST` is loopback and `PORT` is not |
| 446 | 4100 (before the database is opened), and opens the copy without migrating it; |
| 447 | - starts only its HTTP server: no app seeding, JWKS refresh, tiering sweep, health job, junk sweep, |
| 448 | clip re-cuts, copy verification, jobs worker, disk guardian, thumbnail cleanup, object purge, |
| 449 | orphan-recording finalize or Events relay; |
| 450 | - writes no file and creates no directory (the storage checks leave `/api/ready`, as do the remote |
| 451 | tiers; it reports `"mode": "drill"`); |
| 452 | - never opens a file path from the database: every byte route (`/v` and `/c` bytes, `/t`, `/a`, `/f`, |
| 453 | `/o`, `/p/:slug/screenshot`, `/live/:sel/frame.jpg`) answers 503; |
| 454 | watch pages and the `/browse` index still render from the copy; |
| 455 | - connects to nothing and runs no program but `git` (no ffmpeg or ffprobe), opens no UDP socket and |
| 456 | listens on nothing but its port; |
| 457 | - answers 403 to every method but GET, HEAD and OPTIONS, and 503 to `/auth/*`. |
| 458 | |
| 459 | `test/drill-mode.test.js` boots the real server that way against a restored copy |
| 460 | and checks each point. |
| 461 | |
| 462 | ## N-1: the previous release against this one |
| 463 | |
| 464 | `test/n-1.test.js` (roadmap WS-P task 11, in `npm test`) boots this checkout in the drill sandbox |
| 465 | (with writes let through) on an embedded PGlite database and replays `test/fixtures/n-1/`: every call the previous release's clients make (the |
| 466 | openvibe-sdk media and objects clients with an app key, the object explorer, and every player URL, |
| 467 | link, script and form of the `/`, `/browse`, `/v`, `/c`, `/p`, `/me` and `/updates` pages it served), |
| 468 | each answered compatibly (status, JSON, the response fields the client reads). Every PostgreSQL migration |
| 469 | the previous release ran must still be present and unchanged. After each deploy, record the |
| 470 | release now in production as the next N-1 (`npm run n-1:record [ref]`) and commit the fixtures. |
| 471 | |
| 472 | ## Client addresses |
| 473 | |
| 474 | Express `trust proxy` is `loopback` (`server/client-ip.js`): `req.ip` is the `X-Forwarded-For` that the |
| 475 | local nginx set when the request came through it, and the socket address for anyone else. A caller |
| 476 | that reaches port 4100 directly cannot choose its IP by sending `X-Forwarded-For` or |
| 477 | `CF-Connecting-IP` itself. View counting and the live-frame rate limit read `req.ip`, |
| 478 | never the headers (`test/trust-proxy.test.js`). |
| 479 | |
| 480 | ## Segment-native video settings |
| 481 | |
| 482 | | Environment variable | Default | Purpose | |
| 483 | |---|---:|---| |
| 484 | | `MEDIA_HLS_ENABLED` | `0` | Enable timeline generation and HLS routes. | |
| 485 | | `MEDIA_CMAF_TRANSCODE` | `0` | Encode incompatible source streams to H.264/AAC for the source timeline; uses CPU. | |
| 486 | | `MEDIA_CMAF_TRANSCODE_MAX_SECONDS` | `14400` | Maximum duration of a source needing CMAF transcoding (four hours). | |
| 487 | | `MEDIA_CMAF_TRANSCODE_THREADS` | `2` | libx264 threads for one CMAF transcode, which also runs at nice 15 so live streams keep the CPU. | |
| 488 | | `MEDIA_TIMELINE_LOCAL_MAX_BYTES` | `10737418240` | Maximum local timeline cache (10 GiB; `0` keeps no durable copies locally). | |
| 489 | | `MEDIA_TIMELINE_LOCAL_MIN_FREE_BYTES` | `8589934592` | Below this free disk level, new verified timeline bytes remain durable only. | |
| 490 | | `MEDIA_HLS_BACKFILL_PER_HOUR` | `20` | Maximum lazy `object.cmaf` queues for older VODs per rolling hour on a single Media host; finalize is exempt. | |
| 491 | | `MEDIA_JOBS_HEAVY_CONCURRENCY` | `1` | Keep at 1 while enabling HLS in production. | |
| 492 | |
| 493 | The hourly sweep evicts the least recently served verified local timeline files. A segment whose local copy was |
| 494 | evicted is served from its durable provider. Watch `timeline_cache` on `/api/ready` and job durations during backfill |
| 495 | (docs/media-fabric.md §3). |
| 496 | |
| 497 | ## Per-actor limits |
| 498 | |
| 499 | nginx limits by address; the capability routes also limit by who calls (`server/actor-limits.js`, |
| 500 | openvibe-sdk/limits, roadmap WS-R task 4): the Network token's principal (`svc:community`, `app:app_…`), |
| 501 | the person signed in to `/api/v2/me`, or an app acting for one of its users (`X-OV-User-Id`). An app's |
| 502 | own API key is not counted (a first-party server speaks for all its visitors); a browser with only an |
| 503 | upload token is counted by address. Past a limit: 429 problem+json `rate_limited` with `Retry-After`, |
| 504 | before the route does any work, and one `[Limits]` log line and `media_rate_limited_total{limit,window}`. |
| 505 | |
| 506 | | Routes | Per caller | |
| 507 | |---|---| |
| 508 | | Reads: objects, files, jobs, namespaces, `/api/v2/me` | `MEDIA_LIMITS_MINUTE` / `MEDIA_LIMITS_HOUR` (120 / 3000) | |
| 509 | | Object init, `/upload-url`, `/multipart`, multipart complete; file upload | 30 a minute, 600 an hour | |
| 510 | | Object content PUT, complete; object and file delete, restore, multipart abort; job cancel | 60 / 1200 | |
| 511 | | Multipart parts | 600 / 20000 (the session already bounds them) | |
| 512 | | Job create, approve | 10 / 200 | |
| 513 | | `POST /api/v2/me/ops/recompute` | 6 / 60 | |
| 514 | |
| 515 | Never limited: `/healthz`, `/api/ready`, `/release.json`, `/metrics`, the signed `/internal/events` deliveries. |
| 516 | Counters are per process (a restart forgets them). `test/actor-limits.test.js`. |
| 517 | |
| 518 | ## Usage readings → Billing |
| 519 | |
| 520 | Media's storage usage goes to OpenVibe.Billing as one `platform.usage-sample@1` reading per (project, subject, |
| 521 | tier) where a project's work is billable (`server/billing.js`, plan T5 step 14): |
| 522 | |
| 523 | | metric (`resource`) | what | period | unit | provider | |
| 524 | |---|---|---|---|---| |
| 525 | | `gb-month` | bytes stored: every present copy in `media_locations`, summed by tier, counted once when more than one object names a location and billed to the oldest owner, as that day's share of a GB-month | closed UTC day | `GB` | the tier (`local`/`b2`/`r2`) | |
| 526 | |
| 527 | **Media emits `gb-month` only.** `gib-delivered` is deliberately not emitted: `demand.record()` counts `route()` |
| 528 | calls, not bytes, so reads × `size_bytes` would bill aborted downloads and range reads in full, would bill |
| 529 | presigned B2/R2 redirects Media never serves, and would miss HLS segment reads. It comes back once responses are |
| 530 | metered by the bytes actually written (`sendSlice`/`streamFileWithRange`) and redirected reads are metered by CDN |
| 531 | logs. Nothing here bills delivery. |
| 532 | |
| 533 | Only the newest closed UTC day is aggregated, because the query reads `media_locations`' current state: a day |
| 534 | missed while the service was down is not billed (a reading never claims today's state for an older day). |
| 535 | |
| 536 | First-party traffic (a tenant with no project) and sandbox tenants are never recorded; an object without an |
| 537 | `owner_subject` is still stored with its project (Billing leaves it unrated until it names a user subject). |
| 538 | The SDK reporter (`openvibe-sdk/usage createUsageReporter`) queues each reading idempotency-keyed in |
| 539 | `billing_readings` inside the aggregation's transaction and relays it to Billing's `POST /api/v1/usage` |
| 540 | (`billing.usage.record`) with Media's Network service token (`clientId` `media`, audience `openvibe.billing`; |
| 541 | OpenVibe.Network grants exactly that). A relay that cannot reach Billing (no grant, a missing route, a timeout, |
| 542 | backpressure, a 401/403/404/408/425/429 or a 5xx) stays pending with backoff across restarts; only Billing |
| 543 | refusing the reading itself (any other 4xx) is marked rejected and never retried, so nothing is billed twice. |
| 544 | A stored reading is never edited (a trigger). Re-aggregating a day inserts nothing |
| 545 | (`billing_periods`). |
| 546 | |
| 547 | **Off by default.** `MEDIA_BILLING_INTERVAL_MS=0` (the default) starts no timer and aggregates nothing; a |
| 548 | restore drill starts nothing at all. Turn it on with `MEDIA_BILLING_INTERVAL_MS` plus `OV_BILLING_URL` and, |
| 549 | only if it differs, `OV_BILLING_AUDIENCE`; without a billing URL or `OV_OAUTH_CLIENT_SECRET` readings are |
| 550 | aggregated and stay queued. The runbook is `docs/cutover-media-billing-readings.md`; `test/billing.test.js`. |
| 551 | |
| 552 | ## Health, readiness and metrics |
| 553 | |
| 554 | | path | behavior | |
| 555 | |---|---| |
| 556 | | `GET /healthz` | liveness: the process answers (unchanged; it checks nothing else) | |
| 557 | | `GET /api/ready` | readiness from real checks, each with `status`, `required`, `latency_ms`, `checked_at`. **Required** (503 when one fails): `db` (a real round trip that names the store, postgresql or pglite, and counts the tenants), `valkey` optional (skipped without `VALKEY_URL`), `storage_vods`, `storage_clips`, `storage_pastes`, `storage_thumbnails`, `storage_files`, `storage_objects` (a probe file is written and removed). **Optional** (listed in `degraded`, still 200): `network_jwks`, `remote_b2` / `remote_r2`, `events_outbox`, and `timeline_cache` (bytes vs budget; free disk vs floor). Also `recordings_in_progress` (ffmpeg ingest), `chunked_recordings_in_progress` (browser MediaRecorder sessions that received a chunk in the last five minutes), `recordings_untracked_in_progress` (a recording the database still holds — `vods.is_recording = 1` — that no session in memory knows and whose file is still growing: a chunked upload that resumed after a restart) and `uploads_in_progress` (an object upload whose bytes are moving: a single-part PUT, a multipart part, the assembly of a multipart upload). The three later fields are additive, so a reader that predates one keeps using the others; a count that cannot be answered is `null` (unknown), never zero. | |
| 558 | | `GET /metrics` | Prometheus text (openvibe-shared/metrics) for **direct loopback callers only**; 404 through nginx. HTTP golden signals by route template, process metrics, `release_info`, `release_client_updates_total{outcome,reason}`, plus `media_recordings_in_progress`, `media_timeline_local_bytes`, `media_timeline_local_evictions`, `media_object_locations{provider,state}`, `media_objects{lifecycle_status}`, `media_events_outbox{status}` (only while the outbox runs), `media_job_stale_completions_total{action}` (job writes refused by the lease-token fence). | |
| 559 | | `GET /release.json` | release manifest (`registry.release-manifest@1` 1.1.0, openvibe-shared/release, ADR-016): deployed commit, package versions, components, contract ranges, `metrics_url`. The shared navbar's release-watch polls it. No components are declared, so every release still prompts open tabs to reload. | |
| 560 | | `POST /release-metrics` | open tabs' update reports (release-watch beacons, at most 4 KB; 30 a minute per connecting address, which behind nginx is the proxy, so one budget for all tabs; Sec-GPC/DNT dropped) into `release_client_updates_total` on `/metrics`. 403 in a restore drill, like every write. | |
| 561 | |
| 562 | ## Public serving (no auth unless the item is private) |
| 563 | |
| 564 | | path | behavior | |
| 565 | |---|---| |
| 566 | | `GET /v/:id` | VOD playback — local stream with Range support, live-DVR `.seekable` sidecar while recording, or **302** to a presigned B2/R2 URL. `X-Robots-Tag: noindex`. By numeric id only: the file-name form was retired with C-75 (2026-10-11) | |
| 567 | | `GET /c/:id` | clip playback, same logic, `noindex` | |
| 568 | | `GET /p/:slug` | **301** to OpenVibe.Community, the canonical home for pastes since 2026-09-22 (every slug, found or not) | |
| 569 | | `GET /p/:slug/raw` | **301** to Community's raw text | |
| 570 | | `GET /p/:slug/screenshot` | 301 to a ready, nonprivate legacy screenshot or avatar object; otherwise 301 to Community | |
| 571 | | `GET /t/:id` | thumbnail (id = filename), `noindex`. **`?w=` and/or `?fmt=`** answer a variant (Media Fabric F4, `server/thumbnails/variants.js`): the width snaps up to 64, 96, 160, 320, 480 or 640 (never wider than the source), the format is `webp` (the default), `avif` or `jpeg`, named in the URL and never negotiated (Cloudflare ignores `Vary`). Made on the first request and kept in a bounded cache (`THUMBNAIL_VARIANTS_PATH`, default `thumbnail-variants` beside the thumbnails; `MEDIA_THUMB_VARIANTS_MAX_BYTES` 512 MiB, least recently used dropped; `MEDIA_THUMB_VARIANTS_CONCURRENCY` 2). The key holds the source's modification time, so a rewritten live thumbnail gets a fresh variant. Same caching as the original; a busy cache answers the original with `max-age=60`; a bad `w` or `fmt` is a 400. A 640 px VOD thumbnail of about 24 KB is about 13 KB as a 480 px WebP | |
| 572 | | `GET /f/:key` | file with stored Content-Type + Range, `noindex` | |
| 573 | | `GET /live/:sel/transcript.json` | **transcript + AI timeline API** — full audio-transcription log and AI overview timeline. `:sel` = slot id / slot slug → slot-scoped sessions; **`@username`** → user-scoped (all their slots, works while offline). Returns `{ live, current, sessions[] (each: ai_overview, transcript, duration…), streamer (overview + stream memories), user }`. `?limit=1..50` sessions (default 10), `?app=`. Cached **30s** (that's the rate limit), CORS-open. | |
| 574 | | `GET /live/:sel/chat-insight.json` | **chat insight API** — a user's chat-related AI insight/timeline (`:sel` = `@username` or numeric user id): today-vs-alltime chat overviews, condensed memory, event timeline, plus their streamer overview + stream memories when they stream. Proxied from the app's public chat-AI API over loopback; cached **30s**, CORS-open. | |
| 575 | | `GET /v/:id/transcript.json` | **VOD transcript API** — transcript + AI overview for one existing VOD id (`{ vod_id, title, duration_seconds, ai_overview, transcript, ai_analyzed_at }`). Private VODs → 404. Cached 30s, CORS-open. | |
| 576 | | `GET /o/:id/master.m3u8`, `/o/:id/source/index.m3u8`, `/o/:id/source/{init.mp4,init-head.mp4,init-tail.mp4,NNNNNN.m4s}`, `/o/:id/<rendition>/…` | segment-native video (F3.1), only with `MEDIA_HLS_ENABLED=1` (off by default; off, these 404): HLS playlists written from the object's `media_timeline` rows (VOD, `#EXT-X-ENDLIST`) and its CMAF segments (Range; a segment packed by `object.pack` (F3.3) is a byte range of its chunk). The master playlist lists one variant per rendition whose rows exist — the `source` cut plus, with `MEDIA_RENDITIONS=1`, an on-demand 720p rung cut by `rendition.create` and queued lazily by this route and by `download?format=json` (F4 slice 1, docs/media-fabric.md §5). Same check as `GET /o/:id` (and, like it, no `Access-Control-Allow-Origin` on a private or sandbox answer): private and sandbox objects only with the object's `?exp&sig` (`/api/v2/:app/objects/:id/download?format=json` → `hls_url`, a playlist token valid `MEDIA_HLS_PLAYLIST_TTL_S` (6 h; 60 s to 12 h) that no other route accepts; a download signature works too), carried onto every URI; deleted 410. The `object.cmaf` job makes the timeline; a recording's finalize queues it, and `download?format=json` queues it for an object that has none yet; the v1 VOD and clip answers carry the same `hls_url` (and `hls_expires_at` when signed). A materialized clip (F3.6, `MEDIA_MATERIALIZED_CLIPS=1`) plays its own persisted rows: the playlist carries `#EXT-X-DISCONTINUITY` + a fresh `#EXT-X-MAP` where a re-encoded edge's `init-head.mp4`/`init-tail.mp4` replaces the source init (docs/materialized-clips.md). `noindex` | |
| 577 | | `GET /robots.txt` | crawler policy (`openvibe-shared/seo.robotsTxt`: AI and search crawlers named, `/api/`, `/auth/`, `/internal/`, `/o/`, `/metrics`, `/live/`, `/me` disallowed) and the sitemap | |
| 578 | | `GET /sitemap.xml` | the media index and the watch pages Media is the canonical home of: public, playable VODs and clips of apps other than Live (Live lists its own `/vod` and `/clip` pages), never AI clips, private, unlisted, sandbox, recording or failed items; thumbnails as image entries. Every listed page renders `index, follow` | |
| 579 | | `GET /llms.txt` | a map of the site for language-model crawlers | |
| 580 | | `GET /llms-full.txt` | the same map plus the full text of the index and the newest watch pages Media is the canonical home of (`openvibe-shared/seo.llmsFull`; rendering stops at 512 KB, and any further pages become its "truncated" line) | |
| 581 | | `GET /live/:sel/frame.jpg` | **live frame API** — near-realtime JPEG frame of an actively-live stream slot, extracted from its in-progress recording. `:sel` = slot id (`1`), slot **slug** (`whip`), or **`@username`** (that streamer's top-viewed live slot; slug/username resolve via the app's `/api/streams` listing, cached 5s). Optional `?w=64..1920` scales the width, `?app=` selects the tenant (default `live`; internal base URLs from `APP_INTERNAL_URLS` JSON env or `LIVE_APP_INTERNAL_URL`). Cached **5s per slot** (that cache is the rate limit), CORS-open for external APIs/bots/dashboards. Not live → **404 with a styled OFFLINE card** (real JPEG bytes — dev pipelines decode the body as image/jpeg) so `<img>` embeds degrade nicely (`?format=json` for JSON errors); `503` + card when live but a frame can't be cut. | |
| 582 | |
| 583 | Sitemap entries carry lastmod from the release commit for the index and each watch item's recorded time. |
| 584 | |
| 585 | Watch pages (`/v/:id`, `/c/:id` on a browser navigation) offer a player only when the item's |
| 586 | [readiness](#readiness) is `playable`; otherwise they say why (still being recorded, still processing, |
| 587 | not available) and are `noindex`. They carry a schema.org `VideoObject` when the item is public and |
| 588 | playable (none on unlisted or private pages). AI clips (`auto_generated`) are `noindex, follow` |
| 589 | wherever they belong, labelled "AI clip", attributed to no person (`creator` OpenVibe AI) and |
| 590 | `isBasedOn` their source VOD, as on Live. |
| 591 | |
| 592 | Private items (and legacy rows with no visibility and `is_public = 0`) answer |
| 593 | exactly like a missing id — the same 404 and body — unless the request bears the |
| 594 | owning app's API key, optionally acting for the owner via `X-OV-User-Id`. That holds |
| 595 | for the watch page, the bytes, `transcript.json` and `/p/:slug/raw`; the v1 detail routes (`GET /vods/:id`, `/clips/:id`) |
| 596 | apply it to an app acting for someone other than the owner. A call acting for a user writes |
| 597 | only that user's VODs and clips (update, delete, ingest, chunks, finalize, re-cut, thumbnail): |
| 598 | someone else's private one is the missing answer, anything else 403. The media index lists a |
| 599 | thumbnail only while its VOD or clip is public. Tests: `test/security-*.test.js`. |
| 600 | |
| 601 | `GET /o/:id` serves object bytes by canonical id (`med_…`): public/unlisted |
| 602 | objects openly (and, for a JPEG, PNG, WebP, AVIF or GIF with a local copy, `?w=&fmt=` answers a resized variant exactly |
| 603 | as `/t/:id` does, from its own cache `OBJECT_VARIANTS_PATH`, default `object-variants`; Network's `/avatar/:username?s=` |
| 604 | and the shared navbar ask for avatars this way; a private image takes a variant only with its signature, served |
| 605 | private and no-store like the original, which is how openvibe.download shows its previews; without a local copy, or when the cache is busy, the original), private ones only with a valid signature from |
| 606 | `/api/v2/:app/objects/:id/download`, deleted ones 410 (a deleted private object |
| 607 | without a signature is a 404 like any other private one). A native object in the |
| 608 | R2 popularity cache redirects to its verified R2 copy; a GET of a native object |
| 609 | counts its viewer for the day, identifying no one (see Storage tiering). |
| 610 | |
| 611 | ## Unsafe-image review (VIS-6) |
| 612 | |
| 613 | A visible upload is marked `review_status = 'pending'` and is not served until OpenVibe.Vision (configured by |
| 614 | `VISION_URL` or `OV_VISION_URL`, plus Media's Network service principal) answers. The reviewer sorts it into exactly |
| 615 | three categories: |
| 616 | |
| 617 | | Category | Media's action | |
| 618 | |---|---| |
| 619 | | `fine` | `clear`: the object serves as before | |
| 620 | | `sex_acts` | `hidden`: `visibility = private`; `/o/:id`, its HLS playlists, `public_url` and every public surface close; a signed download still reaches it for its owner | |
| 621 | | `minor_sexual` | `quarantined`: private and 404 even with a signature; Vision's operator report id is kept in `review_report_id`, and an `evidence` hold is placed in the same transaction so the owner's delete or retention purge cannot destroy it | |
| 622 | |
| 623 | If Vision is unreachable, the object stays `pending` (closed), never public. `POST /api/v2/:app/objects/:id/review` |
| 624 | re-runs the review. A category is an operator decision: |
| 625 | `POST /api/v2/:app/objects/:id/review-decision` needs the service capability `media.object.review.decide` (Vision's), |
| 626 | takes `fine | sex_acts | minor_sexual`, and releases the evidence hold only for `fine` or `sex_acts` with |
| 627 | `released_by`/`released_at` on the hold row. `POST /api/v2/:app/objects/:id/operator-view` needs |
| 628 | `media.object.review.view`, refuses a fine or pending object, and answers a 60-second signed URL under a separate |
| 629 | `operator` signature purpose; every view is written to `media_review_views` and a `media.moderation.action` audit |
| 630 | event. `GET /:id` includes `review: { category, status, report_id, hold_id, reviewed_at }`, and `objectPublic`'s |
| 631 | `public_url` is null for a hidden, quarantined or pending object. Rows uploaded before this migration have NULL review |
| 632 | fields and keep serving as they did. Tests: `test/object-review.test.js`. |
| 633 | |
| 634 | ## S3 surface and zones |
| 635 | |
| 636 | Media answers a subset of S3 at `https://s3.openvibe.media` (ADR-031; DNS-only since 2026-10-21, so an object may be larger than Cloudflare's 100 MB; `https://s3.openvibe.network` is the same surface behind Cloudflare). Each bucket is a zone of a project environment, and zones are managed by OpenVibe.Zone. Media issues and verifies the `ovk_` access keys, and every object stays an ordinary Media object. |
| 637 | |
| 638 | See [docs/s3.md](docs/s3.md) for the operations, the rules and the tests. |
| 639 | |
| 640 | ## Object model and API v2 |
| 641 | |
| 642 | Every stored blob is a **media object** (`med_<ULID>`, `media_objects`) with one |
| 643 | `media_locations` row per copy (local / B2 canonical / R2 cache, each `present`, |
| 644 | `missing`, `pending` or `corrupt`). The `vods`, `clips` and `files` rows are typed projections over objects (`object_id` column); every |
| 645 | existing route and response is unchanged, and the old write paths keep the |
| 646 | model current. Full reference: **[docs/object-model.md](docs/object-model.md)**. |
| 647 | |
| 648 | - **Object projection** — C-75 was retired on 2026-10-10 after production drift was 0 of 3036 rows. Every write makes its media object in the same transaction (`server/objects/model.js` `withObject`). Operators retain `scripts/reconcile-objects.js` and `scripts/object-invariant.js` as checks ([details](docs/object-model.md#keeping-the-model-current-object-first-writes)). |
| 649 | - **API** — `/api/v2/:app/objects`: init → `PUT /:id/content` (sha256, size, |
| 650 | quota) → `/:id/complete`; `GET /:id`, cursor `GET /`, soft `DELETE /:id` |
| 651 | (bytes kept `MEDIA_DELETE_RETENTION_DAYS`), `/:id/restore`, `/:id/download` |
| 652 | (302 for public, HMAC-signed short-lived URL for private — |
| 653 | `MEDIA_SIGNING_SECRET`). App key, or a Network service token with |
| 654 | `media.object.upload` / `media.object.read` for namespace `:app`; |
| 655 | `X-OV-Subject` sets the owner. `POST /:id/review` applies a review (see Unsafe-image review). |
| 656 | `POST /:id/rotate { degrees | quarter_turns }` turns a ready JPEG/PNG/WebP picture clockwise by 90, 180 or |
| 657 | 270 degrees; only its owner (the `X-OV-Subject` or app user it belongs to) may. The stored pixels are turned, |
| 658 | the uploader's file is kept beside them as `<file>.original`, and every size follows because the variants are |
| 659 | made from the stored file. Each turn is recorded with who and when in the object's `metadata.rotation_audit`. |
| 660 | A picture whose EXIF did not settle its orientation is marked `orientation_status: orientation unknown`; |
| 661 | with the owner's detector on in OpenVibe.AI (`pics.orientation`; off by default) `server/images/ai-orientation.js` |
| 662 | asks it about a 256 px copy and turns the picture only above the owner's confidence. `node |
| 663 | scripts/orientation-detect.js [--apply] [--all]` walks the unknown pictures in batches (dry run counts, no spend). |
| 664 | - **Retention holds** — `media_holds` (`moderation`, `dmca`, `creator_pin`, |
| 665 | `admin`, `evidence`; reason, placed by/at, released by/at, note; `/:id/holds` |
| 666 | and the staff routes `/api/v1/:app/admin/storage/holds`, app key only): a held |
| 667 | object cannot be deleted by any path (409 / DB trigger) or moved between tiers |
| 668 | (`moveToHot` included), and a clip cut from a held VOD is held too |
| 669 | ([docs](docs/object-model.md#retention-holds)). |
| 670 | - **Reconciliation** — `node scripts/reconcile-objects.js [--verify] [--hash]`: |
| 671 | read-only by default; `--verify` HEADs B2/R2 and records location states. |
| 672 | Reports missing canonical copies, lost local files, size/hash mismatches, |
| 673 | orphan locations, deleted-but-still-served objects and unprojected rows. |
| 674 | - **Orphan report** — monthly job `storage.orphans.scan` (`MEDIA_ORPHAN_SCAN_DAYS`, 30) and |
| 675 | `node scripts/vods-orphans-report.js --storage [--no-remote]`: local files and B2/R2 keys no row |
| 676 | names, recorded copies that are not there, open multipart uploads (Media's and the buckets'). |
| 677 | Report only: it never deletes anything ([docs](docs/object-model.md#storage-orphan-report)). |
| 678 | - **Public object-size invariant** — `MEDIA_PUBLIC_OBJECT_MAX_MB` (500) / |
| 679 | `_TARGET_MB` (256) / `_WARN_MB` (384): v2 refuses oversized public playback |
| 680 | uploads; `node scripts/object-invariant.js` lists offenders and records them in |
| 681 | `media_invariant_violations`. No automatic re-encoding. |
| 682 | |
| 683 | ## Storage tiering |
| 684 | |
| 685 | `vod/vod-storage.js` (ported nearly as-is): local → B2 canonical cold tier → |
| 686 | R2 popularity cache; verified uploads before any deletion; periodic sweep |
| 687 | (offload by age/views/last-access, aggressive drain under disk pressure, R2 |
| 688 | promote/demote); presigned-302 playback. Knobs live in `media_settings` under |
| 689 | `storage_tier.*` (JSON values). Every promotion to and demotion from R2 (the sweep, an admin move, a |
| 690 | clip cut fetching an R2 VOD home, the eviction drill) is logged in `media_tier_decisions` with the |
| 691 | inputs it saw (views, last access, provider, recording, hold, size), the thresholds in force and their |
| 692 | source, the reason and the outcome; `/admin/storage/tiers/policy` shows the policy and the log, and |
| 693 | `media_tier_decisions_24h{target="vod",action,outcome}` counts them on `/metrics` (`test/tier-decisions.test.js`). |
| 694 | |
| 695 | **Native v2 objects** (roadmap WS-G task 11; [docs](docs/object-model.md#tiering-of-native-objects)) keep |
| 696 | their canonical copy and, when popular, get a verified copy in R2 that `/o/:id` plays from. Popularity is |
| 697 | unique viewers per object per UTC day, counted on `/o/:id` without storing an IP address or subject id |
| 698 | (a daily-salted hash, deleted with its salt when the day ends; only the counts stay, 30 days), summed over |
| 699 | 7 days. The policy is the revisioned namespace `media.object_tier`: promote at `promoteMinUniqueViewers7d` |
| 700 | (500) unique viewers, viewed within 3 days, 16 MB to 4 GB, at most 3 per sweep; demote after 14 idle days |
| 701 | or when the object is no longer ready. Its **activation gate `active` is off by default**: the sweep (step 4 |
| 702 | of this one) then moves nothing and records what it would do as `dry_run` decisions. Holds block every |
| 703 | move, only a verified canonical copy is promoted (re-hashed; the R2 copy's size and sha256 are checked |
| 704 | before it is used), and a demotion never removes the last good copy. Every object classifies into one of the six |
| 705 | storage classes (`video`, `image`, `download`, `game-asset`, `attachment`, `backup`): kind and mime type decide |
| 706 | video and image, and an app may declare any class in the object's metadata (`metadata.class`, or `media_class` / |
| 707 | `placement_class`), so each class's budgets, hysteresis band and R2 storage ceiling act on that class alone. |
| 708 | Decisions: |
| 709 | `media_object_tier_decisions`, `/admin/storage/tiers/objects/*`, `/me/ops`, and |
| 710 | `media_tier_decisions_24h{target="object"}` (`test/object-tiering.test.js`). |
| 711 | |
| 712 | The storage engine's CLI: |
| 713 | |
| 714 | ``` |
| 715 | node server/vod/vod-storage.js check|drain [targetPct] |
| 716 | ``` |
| 717 | |
| 718 | Background jobs wired in `index.js`: tiering sweep, VOD health job (probe scan, |
| 719 | master-recovery repair, quarantine + cleanup, orphaned-master sweep), disk |
| 720 | guardian (refuses new recordings when free space is critical), stale live-thumb |
| 721 | cleanup, and an on-boot sweep that finalizes recordings orphaned by an unclean |
| 722 | shutdown. |
| 723 | |
| 724 | ## Not ported |
| 725 | |
| 726 | - **Song-request media queue** (`media/` yt-dlp downloader/queue): deeply |
| 727 | coupled to Live's channels/chat/coins subsystems; Live keeps it local. |
| 728 | - Server-side live thumbnail grabbers (RTMP-FLV / JSMPEG-WS / SFU PlainRTP) |
| 729 | and JSMPEG stdin recording: sources live inside the apps now; apps POST |
| 730 | frames / use RTMP-RTP-chunk ingest instead. |
| 731 | - VOD part rotation + live-recording reconciler: tied to the predecessor's |
| 732 | stream registry; apps re-trigger ingest on their side. |
| 733 | - Comments on VODs/clips, notifications, permissions/moderation ranks: owned |
| 734 | by the apps. |
| 735 | |
| 736 | ## VOD storage tiering & disk safety |
| 737 | |
| 738 | `server/vod/vod-storage.js` keeps the local VOD volume healthy without anyone watching it: |
| 739 | |
| 740 | - **Drain policy** — a drain starts when the disk is above `hotDiskPressurePct` (70%) **or** has less than `minFreeGb` (25 GB) free, and stops only when it is under `localLowWaterPct` (60%) **and** has `targetFreeGb` (40 GB) free. Above `criticalDiskPct` (90%) recordings finished more than `criticalMinAgeHours` (2 h) ago are eligible; otherwise 1 day. Least-recently-watched VODs go first. A pass that frees nothing raises `storage.alert drain_stalled` only when eligible VODs could not be moved, or when free space is under the recorder's `VOD_DISK_WARN_GB` (`recorderWarnGb`, 15 GB); a disk shared with other services that stays above the targets with nothing old enough to move is logged once, when that starts. All values live in `media_settings` as `storage_tier.*` (`PUT /api/v1/:app/admin/storage/tiers/settings`). |
| 741 | - **Only real files are candidates.** Rows without a file (a recording that was refused or never produced output, legacy paths) are quarantined as `health_status='missing_file'` and excluded — they used to fill the candidate list and starve the drain. |
| 742 | - **Cadence** — while a drain is still needed the sweep re-runs every `pressureRetryMs` (2 min), otherwise every `sweepIntervalMs` (15 min). A VOD whose upload failed is skipped for 30 min so one bad file cannot pin the drain. |
| 743 | - **Bounded uploads** — each upload is aborted past `uploadTimeoutFloorMs + size / uploadMinThroughputMBps`; a watchdog logs a sweep that has run for hours. |
| 744 | - **Stall alerts** — after `alertAfterStalledPasses` (2) consecutive pressure passes that freed nothing, and whenever the disk is critical, Media queues a `media.storage.alert` event; `media.storage.recovered` follows once the drain works again. Each pass under pressure logs candidates / uploaded / failed / backed-off / quarantined. |
| 745 | - **Status** — `GET /api/v1/:app/admin/storage` includes `sweep` (last result, next run, stalled flag, needsDrain/critical). |
| 746 | |
| 747 | Run `npm test` for the policy regression tests. |
| 748 | |
| 749 | ## Acceptance |
| 750 | |
| 751 | `npm test` runs every `test/*.test.js` in turn (each process gets a migrated PGlite database, or the |
| 752 | PostgreSQL containers under `npm run test:pg`; local stubs, no network). What they prove includes: tenancy |
| 753 | and developer-project tenants (`app-tenants`, `app-tenant-abuse`); the object |
| 754 | model, multipart, reconcile, verification and tiering (`objects-*`, `object-tiering`, `tier-*`); jobs |
| 755 | with leases and fencing (`jobs*`); VOD finalize, durations and storage policy (`vod-*`); private items |
| 756 | stay private and nobody acts on another's ids (`security-private`, `security-idor`, |
| 757 | `private-not-found`); SSRF, secrets and open redirects (`security-ssrf`, `security-secrets`, |
| 758 | `open-redirect`); client addresses (`trust-proxy`); revocation and account data (`revocations`, |
| 759 | `account-data`); events in the change's transaction (`outbox-transaction`, `object-events`); the |
| 760 | restore-drill mode (`drill-mode`); per-actor limits (`actor-limits`); and the previous release against |
| 761 | this one (`n-1`). `npm run smoke` boots a temp instance end to end. |
| 762 | |
| 763 | ## Security |
| 764 | |
| 765 | Reporting a vulnerability: [SECURITY.md](SECURITY.md). The rules the code keeps: |
| 766 | |
| 767 | - **Auth.** App API keys are stored as sha256 hashes and compared in constant time, valid only for |
| 768 | their own tenant. Network user JWTs and principal tokens are verified offline (RS256, audience |
| 769 | `openvibe.media`); principal tokens are checked per verb and namespace; sandbox tokens reach only |
| 770 | their project's sandbox tenant. |
| 771 | - **Private data.** Anyone who may not see a private item gets the answer a missing id gets |
| 772 | (`test/private-not-found.test.js`); sandbox items need a valid signature; uploads a client labels as HTML, SVG or XML are served as |
| 773 | attachments with `nosniff`; view counts store no identifiers. |
| 774 | - **Network exposure.** `trust proxy` is loopback only, so a direct caller cannot choose its address; |
| 775 | `/metrics` answers direct loopback callers only; `/internal/events` deliveries are signed |
| 776 | (`MEDIA_INBOUND_EVENTS_SECRET`). |
| 777 | - **Egress.** Recording pulls only from allow-listed RTMP servers (`MEDIA_RTMP_PULL_ALLOW`, |
| 778 | `test/rtmp-allowlist.test.js`); avatar |
| 779 | ingestion fetches through the SSRF guard; other calls go to the configured Network, Live, Events, B2 |
| 780 | and R2 endpoints. |
| 781 | - **Secrets.** `OV_OAUTH_CLIENT_SECRET`, `MEDIA_SIGNING_SECRET`, `MEDIA_SECRET`, `MEDIA_B2_*`, `MEDIA_R2_*`, `MEDIA_INBOUND_EVENTS_SECRET` and |
| 782 | `VIEW_HASH_SECRET` (the |
| 783 | secret visitor hashes are derived from; it falls back to `MEDIA_SECRET`) live in |
| 784 | `/etc/openvibe/media.env` (0600), by name only. |
| 785 | |
| 786 | ## Deploy |
| 787 | |
| 788 | Production deploys with `sudo ovhost deploy media` on the host (strategy `git-checkout`: fetch, |
| 789 | fast-forward `/opt/openvibe.media`, install on a lockfile change, restart, wait for `/api/ready`, 60 s). |
| 790 | ovhost refuses the restart while a VOD is recording or an object upload is in flight (`recordings_in_progress`, `chunked_recordings_in_progress`, `recordings_untracked_in_progress` and `uploads_in_progress` on `/api/ready`); `--wait-idle` |
| 791 | holds it until none is. The unit is `openvibe-media.service` ([deploy/systemd/](deploy/systemd/openvibe-media.service), |
| 792 | runs as `ubuntu`, media under `/opt/openvibe.media/data`) on `127.0.0.1:4100`, the env file |
| 793 | `/etc/openvibe/media.env`. The listener belongs to `openvibe-media.socket` (with the |
| 794 | `openvibe-media.service.d/socket.conf` drop-in): requests that arrive during a restart wait for the new process |
| 795 | instead of getting 502s; ovhost restarts the service, never the socket. The database is `ov_media` on the host's data role (`sudo /opt/openvibe.host/roles/data/add-service.sh media` |
| 796 | writes its settings); the release migrates it at boot; nginx serves `openvibe.media` from |
| 797 | [deploy/nginx/openvibe.media.conf](deploy/nginx/openvibe.media.conf). After a deploy, record the N-1 |
| 798 | fixtures (`npm run n-1:record`). |
| 799 | |
| 800 | Rollback: ovhost puts the previous sha back by itself when `/api/ready` does not answer 2xx after the |
| 801 | restart; afterwards `sudo ovhost rollback media --to <sha>`. |
| 802 | |
| 803 | <!-- versions:start --> |
| 804 | - openvibe-contracts: v0.423.0 |
| 805 | - openvibe-sdk: v0.161.0 |
| 806 | - openvibe-shared: v4.241.0 |
| 807 | <!-- versions:end --> |
| 808 |