openvibers/openrestream AGENTS.md
AGENTS.md
| 1 | # AGENTS.md — OpenRestream |
| 2 | |
| 3 | ## Project overview |
| 4 | |
| 5 | OpenRestream (`https://openre.stream`) is the network's ingest and restream authority: a creator goes live |
| 6 | once from OBS or any RTMP/WHIP/MPEG-TS sender and OpenRestream fans the stream out to Twitch, YouTube, |
| 7 | Kick and any RTMP/RTMPS/SRT server, with OpenVibe Live on by default. It was extracted from Live so |
| 8 | transport workers survive API deploys (hazard H1: "restarting Live drops live RTMP streams"). |
| 9 | |
| 10 | - **API/UI:** `server/index.js` (`openre-api`, port `4500`), `server/app.js`, `server/api/v1.js`, |
| 11 | `server/api/actor-limits.js`; pages under `server/ui/`; `server/registry/` (resource index/control). |
| 12 | - **Store:** `server/store/` — `definitions.js` (streams/keys), `sessions.js`, `outputs.js`, |
| 13 | `workers.js`, `recordings.js`, `index.js` (SERIALIZABLE transactions). |
| 14 | - **Workers:** `workers/` — `coordinator.js` (leases, generations, drain, recording requests, event |
| 15 | relay), `rtmp-ingest.js`, `restream-worker.js`, `jsmpeg.js`, `webrtc.js` (WHIP + mediasoup SFU), |
| 16 | `thumbnails.js`, `restream/`. |
| 17 | - **Media:** `server/media-client.js` + `server/store/recordings.js` (Media records/finalises; Live |
| 18 | wants the recording). |
| 19 | - **Ops:** `deploy/` (systemd units, nginx `deploy/nginx/openre.stream.conf`, `deploy/scripts/deploy.sh` |
| 20 | and `workers.sh`), env `/etc/openvibe/openre.env`, database `ov_openre`, documents |
| 21 | `docs/cutover.md`, `SECURITY.md`. |
| 22 | |
| 23 | ## Owns, does not own |
| 24 | |
| 25 | Owns stream (input) definitions with a canonical `usr_…` owner and typed refs to other services' |
| 26 | entities (`live:managed_stream:12`, `live:user:45`); ingest keys (SHA-256 only, shown once, rotation |
| 27 | with immediate or grace revocation); ingest sessions (`starting → live → ending → ended | failed`, with |
| 28 | worker, generation and lease); transport workers and generations; destinations (URL + key sealed with |
| 29 | AES-256-GCM, never returned in full), outputs with health/progress/bounded logs and a per-destination |
| 30 | circuit breaker; recording requests to Media and the Live-id migration map; the lifecycle events |
| 31 | `openre.session.started|ended|failed`, `openre.output.healthy|failed`, `openre.recording.requested`, |
| 32 | `openre.key.rotated`. |
| 33 | |
| 34 | Does not own: channel presentation, discovery, watch pages or the creator dashboard (Live); recording |
| 35 | bytes and finalisation (Media records, finalises, thumbnails, fires `vod.ready`); chat, money, |
| 36 | identity (Network). |
| 37 | |
| 38 | ## Processes and consistency model |
| 39 | |
| 40 | `openre-api` (4500) + `openre-session-coordinator` + `openre-rtmp-ingest@`, `openre-jsmpeg@`, |
| 41 | `openre-webrtc@` and `openre-restream-worker@` share one PostgreSQL database (`ov_openre`, ADR-035) |
| 42 | through PgBouncer; every transaction is SERIALIZABLE. **No process depends on another at runtime:** |
| 43 | workers need only the database; restarting the API never touches a session. Each worker registers as |
| 44 | generation N+1; when it is `ready` the coordinator marks older ones `draining` (deadline |
| 45 | `OPENRE_DRAIN_MAX_MS`, default 24 h). Heartbeats every 2 s renew the worker and its sessions' leases; |
| 46 | a worker silent for `OPENRE_WORKER_LEASE_MS` (15 s) is `lost`, its sessions fail and its outputs go back |
| 47 | to `pending`. An output only ever changes its own row; a dead destination fails and the session stays |
| 48 | live. Valkey holds the API's per-actor limit counters; telemetry is aggregated by route template and |
| 49 | pruned after 14 days. |
| 50 | |
| 51 | ## Auth |
| 52 | |
| 53 | Services use a Network client-credentials token for audience `openvibe.openre`, verified offline with |
| 54 | JWKS (a key rotation is honoured on an unknown `kid`); each route checks one capability: |
| 55 | `openre.stream.read/write`, `openre.key.rotate`, `openre.session.read/end`, `openre.output.read/write`, |
| 56 | `openre.resource.read`. A service may send `X-OV-Subject: usr_…` and is then limited to that owner's |
| 57 | streams. Browsers/owners use the Network user JWT (`ov_token` cookie from `/auth/login`, or Bearer; |
| 58 | service principals and typed tokens such as realtime tickets/FedCM assertions are refused) and act on |
| 59 | their own streams; Network role `admin` is staff. UI forms carry a CSRF token derived from the session |
| 60 | cookie; cookies are SameSite=Lax. |
| 61 | |
| 62 | ## Secrets and security |
| 63 | |
| 64 | Ingest keys are 256-bit (`ork_` + 43 base64url), SHA-256 at rest, compared by indexed lookup inside the |
| 65 | RTMP handshake, never logged (last four only), never in an event or a URL after the publish is renamed |
| 66 | to the session id. Destination keys and SRT passphrases are AES-256-GCM (`OPENRE_SECRETS_KEY`, rotation |
| 67 | via `OPENRE_SECRETS_KEY_PREVIOUS`); APIs return `****` + last four; ffmpeg command lines are redacted. |
| 68 | Destination URLs must be `rtmp://`, `rtmps://` or `srt://` with a hostname, no credentials and no |
| 69 | loopback/private/link-local/CGNAT/multicast host — checked on write and again with DNS resolution right |
| 70 | before ffmpeg starts (`OPENRE_DEST_ALLOW_PRIVATE` is the LAN-box opt-out). The RTMP public listener |
| 71 | accepts publish only (play refused); loopback play/FLV ports accept only `/live/ses_<ULID>` paths. |
| 72 | `/internal/` answers only on loopback; deliveries carry a valid v2 signature. |
| 73 | |
| 74 | ## Deploy |
| 75 | |
| 76 | Release layout: `/opt/openre.stream/repo`, `/opt/openre.stream/releases/<sha12>` (full checkout + |
| 77 | `node_modules`, owned by ubuntu), `current` → the release the API and coordinator run; env |
| 78 | `/etc/openvibe/openre.env`; store `ov_openre`; units in `deploy/systemd/`; nginx |
| 79 | `deploy/nginx/openre.stream.conf`; script `deploy/scripts/deploy.sh`. `sudo ovhost deploy openre` is |
| 80 | the release+API path (strategy `release-layout`); the wrapper keeps the old subcommands |
| 81 | (`release <ref>`, `api [sha]`, `rollback [sha]`, `plan`, `workers`, `status`). An API restart is refused |
| 82 | while an ingest session is open (`--wait-idle` holds, `--force` overrides); an API deploy never |
| 83 | restarts a worker. A worker deploy starts the `@<sha>` instances; older ones are disabled and drain |
| 84 | (never `systemctl restart` a worker during a broadcast; `systemctl stop` drains up to 30 min). ovhost |
| 85 | rolls `current` back when the API is not ready in 60 s; workers roll back by starting the previous |
| 86 | generation. Restore drills use `OPENRE_DRILL=1` (API reads only; coordinator/workers exit; relay/Media |
| 87 | off) — never set it in the production env. Ports: 1936 RTMP, 9736 JSMPEG, WHIP on |
| 88 | `ingest.openre.stream`, 4500 API, 10200–10300/udp mediasoup; `OPENRE_RTMP_EXTRA_PORTS=1935` after Live |
| 89 | retires its listener. |
| 90 | |
| 91 | ## Commands and tests |
| 92 | |
| 93 | ```bash |
| 94 | npm ci |
| 95 | npm run dev # openre-api on http://127.0.0.1:4500 |
| 96 | npm run coordinator # each in its own terminal |
| 97 | npm run rtmp-ingest |
| 98 | npm run restream-worker |
| 99 | npm test # every test/*.test.js; RTMP/JSMPEG e2e need ffmpeg |
| 100 | npm run test:pg # the same through PostgreSQL + PgBouncer |
| 101 | npm run migrate-from-live # the Live-id migration map |
| 102 | ``` |
| 103 | |
| 104 | `test/run.js` uses `openvibe-shared/test-runner` (180 s, parallel 3); temp PGlite, generated RSA keys |
| 105 | and stub subscribers, never the network. Coverage includes keys/rotation, sessions and the |
| 106 | coordinator, generations/drain/leases, destinations and logs, the RTMP/JSMPEG/WebRTC end-to-end paths |
| 107 | (real ffmpeg/mediasoup; skipped without them), API restart during a session, drill mode, deploy |
| 108 | wrapper/conf, discovery, limits and the pages. |
| 109 | |
| 110 | ## Conventions and pitfalls |
| 111 | |
| 112 | - CommonJS, 4-space indent, single quotes, semicolons; RFC 9457 problem+json; escaped HTML. |
| 113 | - **Never restart or stop a worker instance during a broadcast.** Deploy a new generation and let the |
| 114 | coordinator drain the old one; an encoder reconnects, a restream/output is handed over, nothing is |
| 115 | killed. `systemctl restart openre-*` is the wrong reflex. |
| 116 | - **Generation and lease ordering are the correctness core:** admit a publish only through the store |
| 117 | (SERIALIZABLE), renew the worker and every owned session/lease in one heartbeat transaction, and let |
| 118 | a stale generation lose atomically. A "fix" that writes a session or output outside its owner is a bug. |
| 119 | - **An output changes only its own row.** A bad destination must not end the session; fail that output, |
| 120 | cool it down, keep the ingest alive. |
| 121 | - **No key in a log, event or play URL.** The publish path is renamed to the session id; the last four |
| 122 | characters are the only hint allowed. |
| 123 | - **SSRF is checked twice:** on write and again with DNS right before ffmpeg; a redirect/DNS change must |
| 124 | not turn a public host into loopback. |
| 125 | - **Recording is a request:** Media records and finalises; OpenRestream never holds the bytes and never |
| 126 | fires `vod.ready` itself. JSMPEG has no recording — say so. |
| 127 | - **The schema only adds**, so a worker on an older release reads a newer database; do not add a |
| 128 | migration that a draining generation cannot run against. |
| 129 |