openvibers/openvibe.media AGENTS.md

AGENTS.md

main 13.5 KB · 174 lines · markdown

Raw HistoryBlame Repository
1# AGENTS.md — OpenVibe.Media
2
3## Project Overview
4
5OpenVibe.Media is the network's standalone multi-tenant media service (port 4100): VOD ingest/recording, storage
6tiering (local → Backblaze B2 → Cloudflare R2), clips, generic files, avatars/attachments, thumbnails, the canonical
7object model (`/api/v2`: objects, locations, variants, holds, derivatives, lifecycle, namespaces), the S3/zone
8surface, and public serving, for every OpenVibe app and developer project.
9
10- **Domain:** `openvibe.media` (plus `s3.openvibe.media`) · **Port:** 4100 · **Service id:** `media` ·
11 **Env prefix:** `MEDIA`/`OV_`
12- **Runtime:** Node ≥22, Express, PostgreSQL 18 through `openvibe-sdk/db` (PGlite in development/tests), Valkey 9,
13 `ffmpeg`/`ffprobe` on PATH, S3-compatible B2/R2 when configured.
14- **Read first:** [README.md](README.md) (long and authoritative), [docs/object-model.md](docs/object-model.md),
15 [docs/media-fabric.md](docs/media-fabric.md), [docs/s3.md](docs/s3.md), [docs/recordings.md](docs/recordings.md),
16 [.env.example](.env.example).
17
18**Network context:** identity, JWKS, visitor sign-in (`media` OAuth client), `identity.subject.resolve` and
19account export/deletion are OpenVibe.Network. OpenVibe.Live answers lineage (`live.lineage.resolve`); Live owns
20channels/streams and who a VOD belongs to. Events are OpenVibe.Events (outbox; **no webhooks since 2026-10-11**).
21AI reads recordings through signed URLs; Media does not analyse them. Pastes moved to OpenVibe.Community and Media
22keeps only legacy screenshot files as media objects.
23
24## Commands
25
26```bash
27npm ci # install (contracts/sdk/shared pinned by tag)
28npm run init-db # node server/db/init.js
29npm start # node server/index.js (ffmpeg/ffprobe required)
30npm test # every test/*.test.js, each on a migrated PGlite, local stubs, no network
31npm run test:pg # the same through PostgreSQL containers
32npm run smoke # scripts/smoke-test.sh: boots a temp instance end to end
33npm run n-1:record [ref] # after a deploy, record the release in production as the next N-1
34node --check <file.js> # syntax check (no linter configured)
35```
36
37No build step. There is no site bundle: public pages/explorer are server-rendered and served by the app.
38
39## Architecture at a Glance
40
41- **Entry:** [server/index.js](server/index.js) — boot/migrations, route mounts, app seeding, JWKS, background jobs
42 (tiering sweep, health scan, junk sweep, clip re-cuts, copy verification, jobs worker, disk guardian, thumbnail
43 cleanup, object purge, orphan finalize, Events relay) and graceful shutdown.
44- **Config/db:** [server/config.js](server/config.js), [server/db/init.js](server/db/init.js),
45 [server/db/database.js](server/db/database.js) (async `openvibe-sdk/db`, every query scoped by `app_id`).
46- **Auth/tenancy:** [server/auth.js](server/auth.js) — app API key (constant-time sha256, own tenant only), Network
47 user JWT (offline JWKS, audience `openvibe.media`, CORS origins), Network principal token (per verb and namespace);
48 first-party tenants seeded token-only (`live`, `community`, `tools`, `games`, `wiki`, `blog`, `inventory`,
49 `media-hub`); developer-project tenants (ADR-014) created on first use, production `prj_…` and sandbox
50 `prj_…-sandbox`, with their own quotas and no public URLs.
51- **VODs/clips:** [server/vod/](server/vod/) — `recorder.js` (RTMP pull/RTP), `media-tools.js`, `finalize.js`,
52 `routes.js`, `clip-cutter.js`, `clips-routes.js`, `vod-storage.js`, `health-scanner.js`, `health-job.js`.
53- **Objects/timeline/jobs:** [server/objects/](server/objects/) (`model.js`, `routes.js`, `timeline.js`,
54 `multipart.js`, `signing.js`, `tiering.js`, `tier-policy.js`, `reconcile.js`, `invariant.js`, `readiness.js`,
55 `review.js`, `namespaces.js`), [server/jobs/](server/jobs/) (queue/worker/leases/types, CMAF/pack/rendition/clip
56 cut/finalize/duration reconcile/storage orphans/move cleanup).
57- **Files/attachments/avatars/images:** [server/files/routes.js](server/files/routes.js),
58 [server/attachments/](server/attachments/), [server/avatars/ingest.js](server/avatars/ingest.js),
59 [server/images/](server/images/), [server/thumbnails/](server/thumbnails/).
60- **Public/private and explorer:** [server/public/routes.js](server/public/routes.js), [server/me/](server/me/)
61 (explorer `/me`, `/api/v2/me`, `/me/ops`), [server/admin/routes.js](server/admin/routes.js),
62 [server/registry/](server/registry/) (authority resource index + control, ADR-048), [server/billing.js](server/billing.js),
63 [server/announce.js](server/announce.js), [server/events.js](server/events.js), [server/account-data.js](server/account-data.js),
64 [server/drill.js](server/drill.js), [server/client-ip.js](server/client-ip.js).
65- **Database:** [migrations/](migrations/) — `apps`, `vods`, `clips`, `files`, `assets`, `content_views`/`content_visits`,
66 `media_settings`, `media_objects/locations/relationships/variants/holds/jobs/invariant_violations`, `media_timeline`,
67 `media_uploads/upload_parts`, `media_recordings/recording_parts`, `media_attachments`, `media_avatar_refs`,
68 `media_access_keys`, `media_zones`, `media_tier_decisions`, `media_verify_runs`, `media_review_views`,
69 `billing_readings/periods`, share tables, `token_revocations`, `subject_merges`, `event_outbox`,
70 `account_data_events`, `telemetry_samples`. The legacy paste tables are dropped (`0007`); nothing may depend on
71 them.
72- **Deploy:** [deploy/systemd/openvibe-media.service](deploy/systemd/openvibe-media.service) (127.0.0.1:4100,
73 `/etc/openvibe/media.env`, `/opt/openvibe.media`, media under `/opt/openvibe.media/data`), the socket unit and
74 drop-in (requests wait across a restart), and [deploy/nginx/openvibe.media.conf](deploy/nginx/openvibe.media.conf).
75 `sudo ovhost deploy media` (git-checkout, database `ov_media`, migrations at boot); it refuses a restart while a VOD
76 records or an upload is in flight — use `--wait-idle`. Roll back with `sudo ovhost rollback media --to <sha>`.
77
78## Ownership and Boundaries
79
80Media owns VODs/recordings, clips and their cuts, files, avatars/attachments, thumbnails, the object model and its
81holds/derivatives/lifecycle, timelines, namespaces/quotas, tenants and project tenants, S3 access keys and zones,
82storage tier decisions, verification runs, public serving and usage readings. It does **not** own pastes (Community),
83channels/streams/VOD lineage (Live), identity/accounts (Network), AI analysis (AI), event delivery (Events), or the
84host/database (Host). Private data: anyone who may not see an item gets the same answer as a missing id.
85
86## Tenancy, Auth and Namespaces
87
88- `:app` is the tenant. Credentials: (1) an app API key, valid only for its own `:app` (403 otherwise); (2) a Network
89 user JWT on browser endpoints, offline-verified, audience `openvibe.media`, `Origin` allowed by the tenant; (3) a
90 Network principal token (`svc:`/`app:`/`mod:`), accepted only on verb routes and checked per namespace `ns`:
91 read `media.object.read`, list `media.object.list`, write `media.object.upload`, delete `media.object.delete`,
92 transform `media.derivative.create`.
93- Developer-project tokens reach only their project's tenant (`prj_<ULID>` production, `prj_<ULID>-sandbox` sandbox);
94 a sandbox token on any other route is 401 `token.sandbox_refused`. Sandbox content is **never public**: `/f/:key`
95 and `/o/:id` answer 404 without a valid signature, `public_url` is null, `/download` always signs, and no platform
96 events are emitted.
97- Capabilities (audience `openvibe.media`): `media.object.upload|read|list|delete`, `media.upload.create`,
98 `media.derivative.create|read`, `media.lifecycle.read|transition`, `media.resource.read|control` (first-party
99 authority index/control), `media.object.s3` (an access key), `media.access_key.manage`, `media.zone.provision`.
100 Outbound: `live.lineage.resolve`, `identity.subject.resolve`, `network.account.export.contribute|deletion.confirm`,
101 `events.event.publish|subscription.manage`.
102- Bearer secrets live only in `/etc/openvibe/media.env` (0600) and appear in code/docs by name only.
103
104## API Surfaces
105
106`/api/v1/:app/...`: VODs (`/vods`, ingest/chunks/finalize), clips, files, thumbnails, assets, admin storage, stats;
107`/api/v2/:app/...`: objects (multipart, lifecycle, holds, derivatives, namespaces), timeline, jobs; `/api/v1/resources`
108(first-party resource index), `POST /api/v1/resources/control`, project access keys/zones; `/api/v2/me` and `/me`;
109public `/v /c /t /f /o`, `/p/:slug` redirects to Community. `/api/health`, `/api/ready`, `/release.json`, `/metrics`
110(loopback), `/internal/events` (signed).
111
112## Conventions
113
114- CommonJS, 4-space indent, single quotes, semicolons; no linter/formatter.
115- New schema = a new `migrations/NNNN_name.sql`; never edit one that has run. Every table is `app_id`-scoped like the
116 queries.
117- **Jobs are leased and fenced:** a run may be retried/superseded; use the lease/fence token, make a type idempotent,
118 and keep the `media.job.*` event in the state change's transaction. A job payload event carries ids/state/ISO
119 times/`has_result` only — never params, result, error text or creator.
120- **Events** are outbox rows in the change's transaction; a replay is one row. No webhooks (C-75).
121- **Storage:** local/B2/R2 copies are verified; a failed copy is never deleted; the disk guardian never drops below
122 its floor; tiering moves are CLI/jobs, not request-time. `MEDIA_DRILL` reads only and writes no byte.
123- **Public serving:** private/sandbox items are 404 (not 403); `nosniff` always; only images (never SVG), video,
124 audio, PDF and plain text inline — everything else `attachment`. View counts store no identifiers.
125- **Client IP:** `trust proxy` is `loopback`; `req.ip` is the nginx-set `X-Forwarded-For` and the socket address
126 otherwise. Never read `X-Forwarded-For`/`CF-Connecting-IP` directly for limits or view counting.
127- **Egress:** recording only from `MEDIA_RTMP_PULL_ALLOW`; avatar ingestion through the SSRF guard; every other call
128 goes to a configured Network/Live/Events/B2/R2 endpoint. Never fetch a user-chosen URL.
129- **N-1:** every client call the previous release made must still answer compatibly, and every migration it ran must
130 still be present unchanged. After a deploy, `npm run n-1:record` and commit the fixtures.
131- Phone first for the explorer/public pages; no Font Awesome; "Calm reduces, only Off removes".
132
133## Key Pitfalls
134
135- **Tenant isolation:** an app key, project token or user JWT must never reach another tenant's rows; a project token
136 names its project in the path and its sandbox cannot be named in a URL. [test/app-tenants.test.js](test/app-tenants.test.js),
137 [test/app-tenant-abuse.test.js](test/app-tenant-abuse.test.js), [test/security-idor.test.js](test/security-idor.test.js).
138- **Private = missing:** keep the 404 answer on every byte/read/list path; a thumbnail of a deleted/private item must
139 not leak. [test/private-not-found.test.js](test/private-not-found.test.js).
140- **Migration 0007 dropped the paste tables** (Pastes moved to Community); never reintroduce `pastes`,
141 `paste_likes`/`paste_comments` or Media's old paste API.
142- **Recordings/uploads gate deploys:** `/api/ready` names `recordings_in_progress`,
143 `chunked_recordings_in_progress`, `recordings_untracked_in_progress`, `uploads_in_progress`; a restart during one
144 is refused. Do not "fix" that by ignoring the check.
145- **Disk safety:** under `VOD_DISK_STOP_GB` the largest running recording is stopped one a minute; only recordings
146 older than a day are offloaded below 90%; under critical running recordings keep writing. Do not add a path that
147 deletes a recording before finalize.
148- **Drill mode** (`MEDIA_DRILL=1`) opens a restored DB copy, loopback/non-4100, no migrations, no seeding, no jobs, no
149 bytes, 503 on byte routes, GET/HEAD/OPTIONS only. [test/drill-mode.test.js](test/drill-mode.test.js).
150- **S3 surface:** an access key is a zone; presigned GET/PUT; never expose another zone's objects or the service's own
151 secrets. See [docs/s3.md](docs/s3.md).
152- **Secrets:** `OV_OAUTH_CLIENT_SECRET`, `MEDIA_SIGNING_SECRET`, `MEDIA_SECRET`, `MEDIA_B2_*`, `MEDIA_R2_*`,
153 `MEDIA_INBOUND_EVENTS_SECRET`, `VIEW_HASH_SECRET` are names only; never log or commit a value.
154
155## Testing
156
157`npm test` runs every file in turn on a migrated PGlite (or the PostgreSQL containers with `npm run test:pg`) with
158local stubs and no network: tenancy/project tenants, object model/multipart/reconcile/verification/tiering, jobs
159leases/fencing, VOD finalize/durations/storage policy, clips/timeline/materialized cuts, private/IDOR/SSRF/secrets/
160redirect, trust proxy, revocations/account data, outbox transaction, drill mode, actor limits and N-1. `npm run smoke`
161boots a temp instance end to end. Run the narrowest file first, then the suite; always `node --check` changed files.
162
163## Documentation
164
165- [README.md](README.md) — purpose, owns/does-not-own, dependencies, capabilities, layout, visitor sign-in, tenancy,
166 API, resource index, drills, N-1, client addresses, segment-native settings, limits, billing readings, health,
167 public serving, review, S3/zones, object model, tiering, acceptance, security, deploy.
168- [docs/object-model.md](docs/object-model.md), [docs/media-fabric.md](docs/media-fabric.md), [docs/s3.md](docs/s3.md),
169 [docs/recordings.md](docs/recordings.md), [docs/materialized-clips.md](docs/materialized-clips.md),
170 [docs/upload-kit.md](docs/upload-kit.md), [docs/player-starts.md](docs/player-starts.md),
171 [docs/startup-tier.md](docs/startup-tier.md), [docs/cutover-media-billing-readings.md](docs/cutover-media-billing-readings.md).
172- [scripts/](scripts/) — smoke, reconcile, invariant, jobs, R2 eviction drill, duration reconcile, orphans report.
173
174