openvibers/openrestream Commits
OpenRe.Stream on PostgreSQL (ADR-035, roadmap WS-X2 G5): openvibe-sdk/db, async throughout, every transaction serializable.
Changes
@@@ -30,6 +30,13 @@ OV_OAUTH_CLIENT_SECRET=#OPENRE_USER_AUDIENCES=openvibe.openre,openvibe.network# ── Store (SQLite, shared by every OpenRe process; WAL) ───────++# PostgreSQL (ADR-035): DATABASE_URL serves through PgBouncer, DATABASE_DIRECT_URL (the owner) runs migrations/ at boot;++# OpenVibe.Host `roles/data/add-service.sh openre` writes both. Every OpenRe process shares the database. Without++# DATABASE_URL, development runs on an embedded PGlite database in data/pglite (OPENRE_PGLITE_DIR): one process only.++DATABASE_URL=++DATABASE_DIRECT_URL=++VALKEY_URL=++# The SQLite file the one-time move reads (scripts/migrate-to-postgres.js).OPENRE_DB_PATH=./data/openre.db# ── Secrets ───────────────────────────────────────────────────@@@ -4,11 +4,14 @@ on:branches: [main]pull_request:jobs:-- # install, node --check, npm test and openvibe-contracts-check: the shared test job (Track Q).++ # install, node --check, npm test and openvibe-contracts-check: the shared test job (Track Q). ADR-035: PostgreSQL++ # 18 + PgBouncer (transaction mode) + Valkey 9 containers (openvibe-sdk scripts/test-services.sh); npm test runs every++ # file on PGlite, test:pg runs them all through PgBouncer, with the tests that start several OpenRe processes.test:-- uses: OpenVibers/OpenVibe.Shared/.github/workflows/test.yml@e8b0caf086ad1f8fceaac80e2dca6cfe64f63602++ uses: OpenVibers/OpenVibe.Shared/.github/workflows/test.yml@423fe519b5fe9594cc81b670e5ba5cb9f21f6176with:apt-packages: ffmpeg # the RTMP end-to-end test publishes and restreams with a real ffmpeg++ test: 'eval "$(node_modules/openvibe-sdk/scripts/test-services.sh up)" && npm test && npm run test:pg'syntax-dirs: server scripts workerscontracts-service: openrecontracts-src: server workers@@@ -32,7 +32,8 @@ The ingest/restream control plane and runtime extracted from OpenVibe.Live. Live- OpenVibe.Network (JWKS for service tokens and user JWTs; OAuth client `openre`; client-credentials tokens for Events/Media)- OpenVibe.Events (event relay; optional: rows wait in the outbox)- OpenVibe.Media (recording requests; optional: sessions work without it)--- OpenVibe.Contracts v0.49.0 (ids, problem+json, capability checks), OpenVibe.SDK v0.12.0 (outbox, token client, per-actor limits), OpenVibe.Shared v1.25.0 (chrome, readiness), pinned by release tarball++- OpenVibe.Contracts v0.76.0 (ids, problem+json, capability checks), OpenVibe.SDK v0.21.3 (`db`, PostgreSQL outbox, token client, per-actor limits), OpenVibe.Shared v1.28.0 (Frame, readiness), pinned by release tarball++- **PostgreSQL 18 and Valkey 9** (OpenVibe.Host `roles/data/`, ADR-035): the store every OpenRe process shares, and the API's per-actor limit counters (optional)- OpenVibe.Live (`live.lineage.resolve`: which channel a Live-linked stream belongs to)- node-media-server 2.7.4 (the RTMP session code Live runs) and the system ffmpeg@@@ -53,7 +54,7 @@ The ingest/restream control plane and runtime extracted from OpenVibe.Live. Live## Processes```-- ┌───────────────── one SQLite database (WAL), /var/lib/openre/openre.db ─────────────────┐++ ┌───────────────── one PostgreSQL database, ov_openre (ADR-035), through PgBouncer ─────────────────┐openre-api (4500) openre-session-coordinator openre-rtmp-ingest@<release> openre-restream-worker@<release>API + UI + /play proxy leases, generations, drain, 0.0.0.0:1936 publish (reuseport) one ffmpeg per output, pulls thedefinitions, keys, output assignment, recording 127.0.0.1 play + HTTP-FLV session's HTTP-FLV from its worker@@@ -64,7 +65,7 @@ openre-api (4500) openre-session-coordinator openre-rtmp-inges- **Generations.** Each worker process registers as generation N+1 of its kind. When a newer generation is `ready`, the coordinator marks older ones `draining` (deadline `OPENRE_DRAIN_MAX_MS`, default 24 h). A draining RTMP worker closes its public listener (the kernel sends new encoder connections to the new generation through the shared port) and keeps its publishers; a draining restream worker keeps its outputs. Idle → the process exits 0 by itself. At the deadline, remaining sessions are ended (encoders reconnect to the newest generation) and remaining outputs are handed over.- **Leases.** Every heartbeat (2 s) renews the worker and the lease of every session it owns in one transaction. A worker silent for `OPENRE_WORKER_LEASE_MS` (15 s) is `lost`: its sessions fail (`openre.session.failed`), its outputs go back to `pending` for another worker. A lost worker that wakes up drops its transports and exits.- **Isolation.** An output only ever changes its own row. A dead destination fails (`openre.output.failed`, destination cooldown 15 min → 24 h) and the session stays live (`test/rtmp-e2e.test.js`, `test/sessions.test.js`, `test/coordinator.test.js`).--- **Store.** SQLite per ADR-007 (single host). Several writer processes is the trigger ADR-007 names for PostgreSQL; all SQL is in `server/store/*`, so that move replaces one layer. Redis/etcd are not used; leases live in the same database.++- **Store.** PostgreSQL 18 through PgBouncer (`ov_openre` on the host's data role, ADR-035; schema in [migrations/](migrations/)), shared by every OpenRe process. Every transaction is SERIALIZABLE (`server/store/index.js`): an ingest worker admitting a publish and the coordinator failing a lost worker's session conflict, and PostgreSQL makes one of them retry, as SQLite's one writer ordered them. Admission is async, so the RTMP worker admits a publish before node-media-server's own publish handling runs. Valkey holds the API's per-actor limit counters. The one-time move is `scripts/migrate-to-postgres.js` (run by OpenVibe.Host `roles/data/switch-service.sh` with every OpenRe unit stopped); `/var/lib/openre/openre.db` stays read-only for 7 days as the rollback.## Running it@@@ -124,7 +125,7 @@ The ids were proposed in [docs/capabilities-proposal/](docs/capabilities-proposa## Events--Written to `event_outbox` in the same SQLite transaction as the change (openvibe-sdk `createOutbox`); the coordinator's relay publishes them with OpenRe's service token. Envelopes validate against `events.event-envelope@1`; `visibility: internal`; no payload ever carries an ingest key or a destination key.++Written to `event_outbox` in the same transaction as the change (openvibe-sdk `createPgOutbox`); the coordinator's relay publishes them with OpenRe's service token. Envelopes validate against `events.event-envelope@1`; `visibility: internal`; no payload ever carries an ingest key or a destination key.| Type | Subject | When ||---|---|---|@@@ -167,7 +168,7 @@ The switch to 1935: once Live no longer listens on 1935, set `OPENRE_RTMP_EXTRA_## Deploying--Layout: `/opt/openre.stream/repo` (the git clone releases are made from), `/opt/openre.stream/releases/<sha12>` (full checkouts with their own `node_modules`, owned by ubuntu), `current` → the release the API and coordinator run, env `/etc/openvibe/openre.env` (see [.env.example](.env.example)), store `/var/lib/openre/openre.db`, units in [deploy/systemd/](deploy/systemd/), nginx [deploy/nginx/openre.stream.conf](deploy/nginx/openre.stream.conf), script [deploy/scripts/deploy.sh](deploy/scripts/deploy.sh). The release and API phases run through `ovhost deploy openre` (OpenVibe.Host, strategy `release-layout`; roadmap WS-N task 11); the script is a thin wrapper that keeps the old subcommands:++Layout: `/opt/openre.stream/repo` (the git clone releases are made from), `/opt/openre.stream/releases/<sha12>` (full checkouts with their own `node_modules`, owned by ubuntu), `current` → the release the API and coordinator run, env `/etc/openvibe/openre.env` (see [.env.example](.env.example)), store `ov_openre` on PostgreSQL (`sudo /opt/openvibe.host/roles/data/add-service.sh openre` writes its settings; the release migrates it at boot; the old SQLite file is `/var/lib/openre/openre.db`), units in [deploy/systemd/](deploy/systemd/), nginx [deploy/nginx/openre.stream.conf](deploy/nginx/openre.stream.conf), script [deploy/scripts/deploy.sh](deploy/scripts/deploy.sh). The release and API phases run through `ovhost deploy openre` (OpenVibe.Host, strategy `release-layout`; roadmap WS-N task 11); the script is a thin wrapper that keeps the old subcommands:```bashsudo deploy/scripts/deploy.sh # ovhost deploy openre: release + api in one@@@ -4,11 +4,11 @@"code": true,"wave": "W7","plan": "OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026)",-- "runtime": "node 22, express, better-sqlite3, node-media-server 2.7.4, ffmpeg",++ "runtime": "node 22, express, PostgreSQL 18 (openvibe-sdk/db, async) + Valkey 9, node-media-server 2.7.4, ffmpeg","port": 4500,"rtmpPort": 1936,"domain": "openre.stream",-- "contracts": "openvibe-contracts v0.49.0 (openvibe-sdk v0.12.0, openvibe-shared v1.25.0)",++ "contracts": "openvibe-contracts v0.76.0 (openvibe-sdk v0.21.3, openvibe-shared v1.28.0)","protocols": {"rtmp": "ingest ported (openre-rtmp-ingest)","restream": "RTMP source to RTMP/RTMPS/SRT destinations ported (openre-restream-worker)",@@@ -16,7 +16,15 @@"webrtc-sfu": "not ported (worker interface only)","jsmpeg": "not ported (worker interface only)"},-- "capabilities": ["openre.stream.read", "openre.stream.write", "openre.key.rotate", "openre.session.read", "openre.session.end", "openre.output.read", "openre.output.write"],++ "capabilities": [++ "openre.stream.read",++ "openre.stream.write",++ "openre.key.rotate",++ "openre.session.read",++ "openre.session.end",++ "openre.output.read",++ "openre.output.write"++ ],"capabilitiesRegistered": true,"livePatch": "applied in Live f0ca18b and deployed; Live wired on 2026-09-23 (OPENRE_URL and the openre.session.* subscription, docs/cutover.md phase A); every Live slot keeps ingest_authority=live until its per-slot cutover (phase B), so no broadcaster uses OpenRe yet","deployed": true,@@@ -2,8 +2,8 @@Description=OpenRe.Stream API and UI (openre-api): definitions, keys, destinations, sessions, playback proxy# The API owns no transport. Restarting it (every API deploy) never touches the transport# workers: they are separate units with no dependency on this one, in either direction.--After=network.target--Wants=openvibe-network.service++After=network.target postgresql.service pgbouncer.service valkey-server.service++Wants=openvibe-network.service postgresql.service pgbouncer.service valkey-server.service[Service]Type=simple@@@ -13,6 +13,7 @@ WorkingDirectory=/opt/openre.stream/currentEnvironmentFile=/etc/openvibe/openre.envEnvironment=NODE_ENV=productionEnvironment=PORT=4500++# The SQLite file the one-time move read (scripts/migrate-to-postgres.js); the store is ov_openre (ADR-035).Environment=OPENRE_DB_PATH=/var/lib/openre/openre.dbEnvironment=PATH=/usr/local/bin:/usr/bin:/binExecStart=/usr/bin/env node server/index.js@@@ -5,7 +5,8 @@ Description=OpenRe.Stream restream worker (ffmpeg per destination), release %i# outputs run until their sessions end (or OPENRE_DRAIN_MAX_MS: then they move over) before it exits 0.# Nothing here depends on openre-api or openre-session-coordinator: an API deploy never restarts# this unit. Never `systemctl restart` it during a broadcast; see README "Deploying".--After=network.target++After=network.target postgresql.service pgbouncer.service valkey-server.service++Wants=postgresql.service pgbouncer.service valkey-server.service[Service]Type=simple@@@ -15,6 +16,7 @@ WorkingDirectory=/opt/openre.stream/releases/%iEnvironmentFile=/etc/openvibe/openre.envEnvironment=NODE_ENV=productionEnvironment=OPENRE_RELEASE=%i++# The SQLite file the one-time move read (scripts/migrate-to-postgres.js); the store is ov_openre (ADR-035).Environment=OPENRE_DB_PATH=/var/lib/openre/openre.dbEnvironment=PATH=/usr/local/bin:/usr/bin:/binExecStart=/usr/bin/env node workers/restream-worker.js@@@ -5,7 +5,8 @@ Description=OpenRe.Stream RTMP ingest worker, release %i# keeps its publishers until they leave (or OPENRE_DRAIN_MAX_MS), and exits 0 by itself.# Nothing here depends on openre-api or openre-session-coordinator: an API deploy never restarts# this unit. Never `systemctl restart` it during a broadcast; see README "Deploying".--After=network.target++After=network.target postgresql.service pgbouncer.service valkey-server.service++Wants=postgresql.service pgbouncer.service valkey-server.service[Service]Type=simple@@@ -15,6 +16,7 @@ WorkingDirectory=/opt/openre.stream/releases/%iEnvironmentFile=/etc/openvibe/openre.envEnvironment=NODE_ENV=productionEnvironment=OPENRE_RELEASE=%i++# The SQLite file the one-time move read (scripts/migrate-to-postgres.js); the store is ov_openre (ADR-035).Environment=OPENRE_DB_PATH=/var/lib/openre/openre.dbEnvironment=PATH=/usr/local/bin:/usr/bin:/binExecStart=/usr/bin/env node workers/rtmp-ingest.js@@@ -1,7 +1,8 @@[Unit]Description=OpenRe.Stream session coordinator: worker leases and generations, output assignment, recording requests, event relay# Stateless between ticks (everything is in the database): restarting it never ends a session.--After=network.target++After=network.target postgresql.service pgbouncer.service valkey-server.service++Wants=postgresql.service pgbouncer.service valkey-server.service[Service]Type=simple@@@ -10,6 +11,7 @@ Group=ubuntuWorkingDirectory=/opt/openre.stream/currentEnvironmentFile=/etc/openvibe/openre.envEnvironment=NODE_ENV=production++# The SQLite file the one-time move read (scripts/migrate-to-postgres.js); the store is ov_openre (ADR-035).Environment=OPENRE_DB_PATH=/var/lib/openre/openre.dbEnvironment=PATH=/usr/local/bin:/usr/bin:/binExecStart=/usr/bin/env node workers/coordinator.jsnnew file mode 100644@@@ -0,0 +1,221 @@++-- phase: expand++-- OpenVibe.OpenRe on PostgreSQL (ADR-035, roadmap WS-X2): the tables as they were on SQLite (converted by openvibe-sdk++-- tools/asyncify/sqlite-schema-to-pg: text COLLATE "C" compares like SQLite, integers are bigint, identities keep their ids),++-- then the openvibe-publishing stores and the openvibe-sdk inbox and outbox. Generated once on 2026-09-28; never edited after it runs.++CREATE TABLE stream_definitions (++ id text COLLATE "C" PRIMARY KEY,++ owner_subject text COLLATE "C" NOT NULL,++ title text COLLATE "C" NOT NULL DEFAULT 'Untitled stream',++ description text COLLATE "C" NOT NULL DEFAULT '',++ protocols text COLLATE "C" NOT NULL DEFAULT '["rtmp"]',++ recording_mode text COLLATE "C" NOT NULL DEFAULT 'vod' CHECK (recording_mode IN ('vod', 'clips', 'none')),++ recording_visibility text COLLATE "C" NOT NULL DEFAULT 'public' CHECK (recording_visibility IN ('public', 'unlisted', 'private')),++ playback_visibility text COLLATE "C" NOT NULL DEFAULT 'public' CHECK (playback_visibility IN ('public', 'unlisted', 'private')),++ mirror_to_live bigint NOT NULL DEFAULT 0,++ state text COLLATE "C" NOT NULL DEFAULT 'active' CHECK (state IN ('active', 'disabled', 'archived')),++ revision bigint NOT NULL DEFAULT 1,++ created_at bigint NOT NULL,++ updated_at bigint NOT NULL++);++++CREATE TABLE external_refs (++ definition_id text COLLATE "C" NOT NULL REFERENCES stream_definitions(id) ON DELETE CASCADE,++ service text COLLATE "C" NOT NULL,++ type text COLLATE "C" NOT NULL,++ ref_id text COLLATE "C" NOT NULL,++ label text COLLATE "C",++ created_at bigint NOT NULL,++ PRIMARY KEY (definition_id, service, type, ref_id)++);++++CREATE TABLE definition_lineage (++ definition_id text COLLATE "C" PRIMARY KEY REFERENCES stream_definitions(id) ON DELETE CASCADE,++ resolution text COLLATE "C" NOT NULL,++ checked_at bigint NOT NULL++);++++CREATE TABLE ingest_keys (++ id text COLLATE "C" PRIMARY KEY,++ definition_id text COLLATE "C" NOT NULL REFERENCES stream_definitions(id) ON DELETE CASCADE,++ key_hash text COLLATE "C" NOT NULL UNIQUE,++ hint text COLLATE "C" NOT NULL,++ status text COLLATE "C" NOT NULL CHECK (status IN ('active', 'grace', 'revoked')),++ created_by text COLLATE "C",++ created_at bigint NOT NULL,++ grace_until bigint,++ revoked_at bigint,++ revoked_reason text COLLATE "C",++ last_used_at bigint++);++++CREATE TABLE workers (++ id text COLLATE "C" PRIMARY KEY,++ kind text COLLATE "C" NOT NULL,++ generation bigint NOT NULL,++ release text COLLATE "C",++ pid bigint,++ host text COLLATE "C",++ state text COLLATE "C" NOT NULL CHECK (state IN ('starting', 'ready', 'draining', 'stopped', 'lost')),++ endpoints text COLLATE "C" NOT NULL DEFAULT '{}',++ started_at bigint NOT NULL,++ ready_at bigint,++ heartbeat_at bigint NOT NULL,++ drain_started_at bigint,++ drain_deadline bigint,++ stopped_at bigint,++ stop_reason text COLLATE "C",++ UNIQUE (kind, generation)++);++++CREATE TABLE ingest_sessions (++ id text COLLATE "C" PRIMARY KEY,++ definition_id text COLLATE "C" NOT NULL REFERENCES stream_definitions(id),++ key_id text COLLATE "C",++ protocol text COLLATE "C" NOT NULL,++ state text COLLATE "C" NOT NULL CHECK (state IN ('starting', 'live', 'ending', 'ended', 'failed')),++ worker_id text COLLATE "C",++ worker_kind text COLLATE "C",++ worker_generation bigint,++ lease_expires_at bigint,++ desired_state text COLLATE "C" NOT NULL DEFAULT 'run' CHECK (desired_state IN ('run', 'end')),++ end_requested_by text COLLATE "C",++ end_reason text COLLATE "C",++ failure_reason text COLLATE "C",++ media_info text COLLATE "C",++ revision bigint NOT NULL DEFAULT 1,++ created_at bigint NOT NULL,++ live_at bigint,++ ending_at bigint,++ ended_at bigint,++ updated_at bigint NOT NULL++);++++CREATE TABLE session_transitions (++ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,++ session_id text COLLATE "C" NOT NULL REFERENCES ingest_sessions(id) ON DELETE CASCADE,++ from_state text COLLATE "C",++ to_state text COLLATE "C" NOT NULL,++ reason text COLLATE "C",++ actor text COLLATE "C",++ at bigint NOT NULL++);++++CREATE TABLE destinations (++ id text COLLATE "C" PRIMARY KEY,++ definition_id text COLLATE "C" NOT NULL REFERENCES stream_definitions(id) ON DELETE CASCADE,++ platform text COLLATE "C" NOT NULL CHECK (platform IN ('youtube', 'twitch', 'kick', 'custom')),++ name text COLLATE "C",++ server_url text COLLATE "C" NOT NULL,++ stream_key_enc text COLLATE "C",++ key_hint text COLLATE "C",++ srt_passphrase_enc text COLLATE "C",++ srt_latency_ms bigint,++ enabled bigint NOT NULL DEFAULT 1,++ auto_start bigint NOT NULL DEFAULT 1,++ quality_preset text COLLATE "C" NOT NULL DEFAULT 'auto',++ custom_video_bitrate bigint,++ custom_audio_bitrate bigint,++ custom_fps bigint,++ custom_encoder_preset text COLLATE "C",++ hold_reason text COLLATE "C",++ consecutive_failures bigint NOT NULL DEFAULT 0,++ cooldown_until bigint,++ last_error text COLLATE "C",++ last_failed_at bigint,++ created_at bigint NOT NULL,++ updated_at bigint NOT NULL++);++++CREATE TABLE outputs (++ id text COLLATE "C" PRIMARY KEY,++ session_id text COLLATE "C" NOT NULL REFERENCES ingest_sessions(id),++ destination_id text COLLATE "C" NOT NULL REFERENCES destinations(id) ON DELETE CASCADE,++ desired text COLLATE "C" NOT NULL DEFAULT 'run' CHECK (desired IN ('run', 'stop')),++ state text COLLATE "C" NOT NULL DEFAULT 'pending' CHECK (state IN ('pending', 'starting', 'live', 'error', 'failed', 'stopped')),++ worker_id text COLLATE "C",++ worker_generation bigint,++ restart_attempts bigint NOT NULL DEFAULT 0,++ next_restart_at bigint,++ ever_live bigint NOT NULL DEFAULT 0,++ last_error text COLLATE "C",++ progress text COLLATE "C",++ started_at bigint,++ live_at bigint,++ ended_at bigint,++ created_at bigint NOT NULL,++ updated_at bigint NOT NULL,++ UNIQUE (session_id, destination_id)++);++++CREATE TABLE output_logs (++ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,++ output_id text COLLATE "C",++ destination_id text COLLATE "C" NOT NULL,++ level text COLLATE "C" NOT NULL CHECK (level IN ('info', 'warn', 'error')),++ message text COLLATE "C" NOT NULL,++ at bigint NOT NULL++);++++CREATE TABLE recordings (++ id text COLLATE "C" PRIMARY KEY,++ session_id text COLLATE "C" NOT NULL UNIQUE REFERENCES ingest_sessions(id),++ mode text COLLATE "C" NOT NULL CHECK (mode IN ('vod', 'clips')),++ state text COLLATE "C" NOT NULL CHECK (state IN ('pending', 'requested', 'recording', 'finalizing', 'finalized', 'failed', 'cancelled')),++ media_app text COLLATE "C",++ media_vod_id text COLLATE "C",++ attempts bigint NOT NULL DEFAULT 0,++ next_attempt_at bigint NOT NULL DEFAULT 0,++ last_error text COLLATE "C",++ created_at bigint NOT NULL,++ updated_at bigint NOT NULL++);++++CREATE TABLE migration_map (++ source_system text COLLATE "C" NOT NULL,++ source_type text COLLATE "C" NOT NULL,++ source_id text COLLATE "C" NOT NULL,++ target_type text COLLATE "C",++ target_id text COLLATE "C",++ status text COLLATE "C" NOT NULL CHECK (status IN ('imported', 'held', 'excluded')),++ reason text COLLATE "C",++ imported_at bigint NOT NULL,++ PRIMARY KEY (source_system, source_type, source_id)++);++++CREATE TABLE leases (++ name text COLLATE "C" PRIMARY KEY,++ holder text COLLATE "C" NOT NULL,++ expires_at bigint NOT NULL++);++++CREATE INDEX idx_definitions_owner ON stream_definitions(owner_subject, state);++CREATE INDEX idx_external_refs_ref ON external_refs(service, type, ref_id);++CREATE INDEX idx_ingest_keys_definition ON ingest_keys(definition_id, status);++CREATE INDEX idx_workers_kind_state ON workers(kind, state, generation);++CREATE INDEX idx_sessions_definition ON ingest_sessions(definition_id, created_at);++CREATE INDEX idx_sessions_state ON ingest_sessions(state);++CREATE INDEX idx_sessions_worker ON ingest_sessions(worker_id, state);++CREATE INDEX idx_transitions_session ON session_transitions(session_id, id);++CREATE INDEX idx_destinations_definition ON destinations(definition_id);++CREATE INDEX idx_outputs_worker ON outputs(worker_id, state);++CREATE INDEX idx_outputs_session ON outputs(session_id);++CREATE INDEX idx_output_logs_output ON output_logs(output_id, id);++CREATE INDEX idx_output_logs_destination ON output_logs(destination_id, id);++CREATE INDEX idx_recordings_state ON recordings(state, next_attempt_at);++++-- openvibe-sdk/events PostgreSQL outbox++CREATE TABLE IF NOT EXISTS event_outbox (++ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,++ event_id text NOT NULL UNIQUE,++ envelope jsonb NOT NULL,++ traceparent text,++ created_at bigint NOT NULL,++ attempts integer NOT NULL DEFAULT 0,++ next_attempt_at bigint NOT NULL DEFAULT 0,++ sent_at bigint,++ seq bigint,++ rejected_at bigint,++ last_error text++);++CREATE INDEX IF NOT EXISTS event_outbox_due ON event_outbox (next_attempt_at, id) WHERE sent_at IS NULL AND rejected_at IS NULL;++CREATE INDEX IF NOT EXISTS event_outbox_sent ON event_outbox (sent_at) WHERE sent_at IS NOT NULL;@@@ -13,15 +13,33 @@"cookie-parser": "^1.4.7","dotenv": "^16.4.7","express": "^4.21.2",++ "iovalkey": "^0.4.0","node-media-server": "2.7.4",-- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.49.0",-- "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.12.0",-- "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.25.0"++ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.76.0",++ "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.21.3",++ "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.28.0",++ "pg": "^8.23.0"++ },++ "devDependencies": {++ "@electric-sql/pglite": "^0.5.8"},"engines": {"node": ">=22.12.0"}},++ "node_modules/@electric-sql/pglite": {++ "version": "0.5.8",++ "resolved": "https://registry.npmjs.org/@electric-sql/pglite/-/pglite-0.5.8.tgz",++ "integrity": "sha512-n9tsbUOhwx2epK1V0ZG9Ar4SHWUju04dhmzZXiSBXwBoleOvIfals33NAaWgagQVAL4Rbvx/Ptsu3P+pA09f6Q==",++ "devOptional": true,++ "license": "Apache-2.0"++ },++ "node_modules/@iovalkey/commands": {++ "version": "0.1.0",++ "resolved": "https://registry.npmjs.org/@iovalkey/commands/-/commands-0.1.0.tgz",++ "integrity": "sha512-/B9W4qKSSITDii5nkBCHyPkIkAi+ealUtr1oqBJsLxjSRLka4pxun2VvMNSmcwgAMxgXtQfl0qRv7TE+udPJzg==",++ "license": "MIT"++ },"node_modules/accepts": {"version": "1.3.8","resolved": "https://registry.npmjs.org/accepts/-/accepts-1.3.8.tgz",@@@ -263,6 +281,15 @@"integrity": "sha512-jJ0bqzaylmJtVnNgzTeSOs8DPavpbYgEr/b0YL8/2GO3xJEhInFmhKMUnEJQjZumK7KXGFhUy89PrsJWlakBVg==","license": "ISC"},++ "node_modules/cluster-key-slot": {++ "version": "1.1.2",++ "resolved": "https://registry.npmjs.org/cluster-key-slot/-/cluster-key-slot-1.1.2.tgz",++ "integrity": "sha512-RMr0FhtfXemyinomL4hrWcYJxmX6deFdCxpJzhDttxgO1+bcCnkk+9drydLVDmAMG7NE6aN/fl4F7ucU/90gAA==",++ "license": "Apache-2.0",++ "engines": {++ "node": ">=0.10.0"++ }++ },"node_modules/color-convert": {"version": "2.0.1","resolved": "https://registry.npmjs.org/color-convert/-/color-convert-2.0.1.tgz",@@@ -372,6 +399,15 @@"node": ">=4.0.0"}},++ "node_modules/denque": {++ "version": "2.1.0",++ "resolved": "https://registry.npmjs.org/denque/-/denque-2.1.0.tgz",++ "integrity": "sha512-HVQE3AAb/pxF8fQAoiqpvg9i3evqug3hoiwakOyZAwJm+6vZehbkYXZ0l4JxS+I3QxM97v5aaRNhj8v5oBhekw==",++ "license": "Apache-2.0",++ "engines": {++ "node": ">=0.10"++ }++ },"node_modules/depd": {"version": "2.0.0","resolved": "https://registry.npmjs.org/depd/-/depd-2.0.0.tgz",@@@ -802,6 +838,49 @@"integrity": "sha512-JV/yugV2uzW5iMRSiZAyDtQd+nxtUnjeLt0acNdw98kKLrvuRVyB80tsREOE7yvGVgalhZ6RNXCmEHkUKBKxew==","license": "ISC"},++ "node_modules/iovalkey": {++ "version": "0.4.0",++ "resolved": "https://registry.npmjs.org/iovalkey/-/iovalkey-0.4.0.tgz",++ "integrity": "sha512-OSUKxJ+s44CLdUTaicX4+pVrBN/zHKIYwr2oKhRDuczr19FwcSCXEZHzcMUYWZNh1mNSCV9bNJHW/T6cHVm9Aw==",++ "license": "MIT",++ "dependencies": {++ "@iovalkey/commands": "^0.1.0",++ "cluster-key-slot": "^1.1.0",++ "debug": "^4.3.4",++ "denque": "^2.1.0",++ "lodash.defaults": "^4.2.0",++ "lodash.isarguments": "^3.1.0",++ "redis-errors": "^1.2.0",++ "redis-parser": "^3.0.0",++ "standard-as-callback": "^2.1.0"++ },++ "engines": {++ "node": ">=18.12.0"++ }++ },++ "node_modules/iovalkey/node_modules/debug": {++ "version": "4.4.3",++ "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz",++ "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==",++ "license": "MIT",++ "dependencies": {++ "ms": "^2.1.3"++ },++ "engines": {++ "node": ">=6.0"++ },++ "peerDependenciesMeta": {++ "supports-color": {++ "optional": true++ }++ }++ },++ "node_modules/iovalkey/node_modules/ms": {++ "version": "2.1.3",++ "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz",++ "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==",++ "license": "MIT"++ },"node_modules/ipaddr.js": {"version": "1.9.1","resolved": "https://registry.npmjs.org/ipaddr.js/-/ipaddr.js-1.9.1.tgz",@@@ -872,12 +951,24 @@"integrity": "sha512-dMInicTPVE8d1e5otfwmmjlxkZoUpiVLwyeTdUsi/Caj/gfzzblBcCE5sRHV/AsjuCmxWrte2TNGSYuCeCq+0Q==","license": "MIT"},++ "node_modules/lodash.defaults": {++ "version": "4.2.0",++ "resolved": "https://registry.npmjs.org/lodash.defaults/-/lodash.defaults-4.2.0.tgz",++ "integrity": "sha512-qjxPLHd3r5DnsdGacqOMU6pb/avJzdh9tFX2ymgoZE27BmjXrNy/y4LoaiTeAb+O3gL8AfpJGtqfX/ae2leYYQ==",++ "license": "MIT"++ },"node_modules/lodash.includes": {"version": "4.3.0","resolved": "https://registry.npmjs.org/lodash.includes/-/lodash.includes-4.3.0.tgz","integrity": "sha512-W3Bx6mdkRTGtlJISOvVD/lbqjTlPPUDTMnlXZFnVwi9NKJ6tiAk6LVdlhZMm17VZisqhKcgzpO5Wz91PCt5b0w==","license": "MIT"},++ "node_modules/lodash.isarguments": {++ "version": "3.1.0",++ "resolved": "https://registry.npmjs.org/lodash.isarguments/-/lodash.isarguments-3.1.0.tgz",++ "integrity": "sha512-chi4NHZlZqZD18a0imDHnZPrDeBbTtVN7GXMwuGdRH9qotxAjYs3aVLKc7zNOG9eddR5Ksd8rvFEBc9SsggPpg==",++ "license": "MIT"++ },"node_modules/lodash.isboolean": {"version": "3.0.3","resolved": "https://registry.npmjs.org/lodash.isboolean/-/lodash.isboolean-3.0.3.tgz",@@@ -1115,9 +1206,9 @@}},"node_modules/openvibe-contracts": {-- "version": "0.49.0",-- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.49.0",-- "integrity": "sha512-lapsLoBm3ZpndgiOMELdCBh2PJtuuyqj6SUhQsfwJkxnLdQfyhXfRFgiuLh+iQrg4kqtG9611191Mp8zAfk7fQ==",++ "version": "0.76.0",++ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.76.0",++ "integrity": "sha512-zTdx776KWrcJKm1JTnf+0+BnsvswdvXy1ldPyse1hSqhnW/SSr98sX696eRetFUhTl8qVM1xSRCykqIUbqItPw==","license": "AGPL-3.0-only","dependencies": {"ajv": "^8.17.1",@@@ -1131,18 +1222,38 @@}},"node_modules/openvibe-sdk": {-- "version": "0.12.0",-- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.12.0",-- "integrity": "sha512-iG1h905O1f7y/PnP8xLA6GCVzXEBDfSaBj2tc1ucNreWg2sT/AJWosb8NW0WRMK5WS4sIQ6tMsprmDLj/fjjeQ==",++ "version": "0.21.3",++ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.21.3",++ "integrity": "sha512-oAcxznSHKko+7zQCHS2VrSW2gqpXgbagnu+UZ60vrW/QjwtoCvatMZvtgEwpekYFGzpI7xklvCe0EOb1fa1H2g==","license": "MIT","engines": {"node": ">=20"++ },++ "peerDependencies": {++ "@electric-sql/pglite": "^0.5.8",++ "better-sqlite3": ">=11",++ "iovalkey": "^0.4.0",++ "pg": "^8.23.0"++ },++ "peerDependenciesMeta": {++ "@electric-sql/pglite": {++ "optional": true++ },++ "better-sqlite3": {++ "optional": true++ },++ "iovalkey": {++ "optional": true++ },++ "pg": {++ "optional": true++ }}},"node_modules/openvibe-shared": {-- "version": "1.25.0",-- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.25.0",-- "integrity": "sha512-oqZ+PvljrWi8IEbtg6bDYai17ZV2sALFx/ezMptniELSr10TgFUiGUQt+b1IWwiHv0T2Y1U2iSrkirJxMNgjSw==",++ "version": "1.28.0",++ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.28.0",++ "integrity": "sha512-WLd11lDG75veVi7mFwngquK2nqIQV86Z14Ad/xE2dYvyMT5mhKJxxxlKUoskpdyLeap0ucMvzzS8Tg7vPGwYoQ==","license": "MIT","dependencies": {"jsonwebtoken": "^9.0.0"@@@ -1163,6 +1274,134 @@"integrity": "sha512-A/AGNMFN3c8bOlvV9RreMdrv7jsmF9XIfDeCd87+I8RNg6s78BhJxMu69NEMHBSJFxKidViTEdruRwEk/WIKqA==","license": "MIT"},++ "node_modules/pg": {++ "version": "8.23.0",++ "resolved": "https://registry.npmjs.org/pg/-/pg-8.23.0.tgz",++ "integrity": "sha512-Ip2EQCngowJLGOfCwkFhPXU7/ljlhn6Rxlmy4XYfL2Y+vyRM59+8uR2xqRWKdYmbXmxCFOAmKxBuSUCdF34qLg==",++ "license": "MIT",++ "dependencies": {++ "pg-connection-string": "^2.14.0",++ "pg-pool": "^3.14.0",++ "pg-protocol": "^1.16.0",++ "pg-types": "2.2.0",++ "pgpass": "1.0.5"++ },++ "engines": {++ "node": ">= 16.0.0"++ },++ "optionalDependencies": {++ "pg-cloudflare": "^1.4.0"++ },++ "peerDependencies": {++ "pg-native": ">=3.0.1"++ },++ "peerDependenciesMeta": {++ "pg-native": {++ "optional": true++ }++ }++ },++ "node_modules/pg-cloudflare": {++ "version": "1.4.0",++ "resolved": "https://registry.npmjs.org/pg-cloudflare/-/pg-cloudflare-1.4.0.tgz",++ "integrity": "sha512-Vo7z/6rrQYxpNRylp4Tlob2elzbh+N/MOQbxFVWCxS7oEx6jF53GTJFxK2WWpKuBRkmiin4Mt+xofFDjx09R0A==",++ "license": "MIT",++ "optional": true++ },++ "node_modules/pg-connection-string": {++ "version": "2.14.0",++ "resolved": "https://registry.npmjs.org/pg-connection-string/-/pg-connection-string-2.14.0.tgz",++ "integrity": "sha512-XwWDGcLRGCXAR8F/AM5bG7Q+A3Wm2s6QeEjlOKZLlH3UYcguiqCWKyWXVag5TLTIjR7oOJUY8kcADaZgWPyLeg==",++ "license": "MIT"++ },++ "node_modules/pg-int8": {++ "version": "1.0.1",++ "resolved": "https://registry.npmjs.org/pg-int8/-/pg-int8-1.0.1.tgz",++ "integrity": "sha512-WCtabS6t3c8SkpDBUlb1kjOs7l66xsGdKpIPZsg4wR+B3+u9UAum2odSsF9tnvxg80h4ZxLWMy4pRjOsFIqQpw==",++ "license": "ISC",++ "engines": {++ "node": ">=4.0.0"++ }++ },++ "node_modules/pg-pool": {++ "version": "3.14.0",++ "resolved": "https://registry.npmjs.org/pg-pool/-/pg-pool-3.14.0.tgz",++ "integrity": "sha512-gKtPkFdQPU3DksooVLi9LsjZxrsBUZIpa+7aVx+LV5pNh0KzP4Zleud2po+ConrxbuXGBJ6Hfer6hdgpIBpBaw==",++ "license": "MIT",++ "peerDependencies": {++ "pg": ">=8.0"++ }++ },++ "node_modules/pg-protocol": {++ "version": "1.16.0",++ "resolved": "https://registry.npmjs.org/pg-protocol/-/pg-protocol-1.16.0.tgz",++ "integrity": "sha512-sILXutLVjCLjcDuOmvhX5e2Z4cS5qG/6Bu3VkpFwdf/633ElGLpEh9bgmuI5I4sqKqkifQiGyiCcx1HdtrK7tg==",++ "license": "MIT"++ },++ "node_modules/pg-types": {++ "version": "2.2.0",++ "resolved": "https://registry.npmjs.org/pg-types/-/pg-types-2.2.0.tgz",++ "integrity": "sha512-qTAAlrEsl8s4OiEQY69wDvcMIdQN6wdz5ojQiOy6YRMuynxenON0O5oCpJI6lshc6scgAY8qvJ2On/p+CXY0GA==",++ "license": "MIT",++ "dependencies": {++ "pg-int8": "1.0.1",++ "postgres-array": "~2.0.0",++ "postgres-bytea": "~1.0.0",++ "postgres-date": "~1.0.4",++ "postgres-interval": "^1.1.0"++ },++ "engines": {++ "node": ">=4"++ }++ },++ "node_modules/pgpass": {++ "version": "1.0.5",++ "resolved": "https://registry.npmjs.org/pgpass/-/pgpass-1.0.5.tgz",++ "integrity": "sha512-FdW9r/jQZhSeohs1Z3sI1yxFQNFvMcnmfuj4WBMUTxOrAyLMaTcE1aAMBiTlbMNaXvBCQuVi0R7hd8udDSP7ug==",++ "license": "MIT",++ "dependencies": {++ "split2": "^4.1.0"++ }++ },++ "node_modules/postgres-array": {++ "version": "2.0.0",++ "resolved": "https://registry.npmjs.org/postgres-array/-/postgres-array-2.0.0.tgz",++ "integrity": "sha512-VpZrUqU5A69eQyW2c5CA1jtLecCsN2U/bD6VilrFDWq5+5UIEVO7nazS3TEcHf1zuPYO/sqGvUvW62g86RXZuA==",++ "license": "MIT",++ "engines": {++ "node": ">=4"++ }++ },++ "node_modules/postgres-bytea": {++ "version": "1.0.1",++ "resolved": "https://registry.npmjs.org/postgres-bytea/-/postgres-bytea-1.0.1.tgz",++ "integrity": "sha512-5+5HqXnsZPE65IJZSMkZtURARZelel2oXUEO8rH83VS/hxH5vv1uHquPg5wZs8yMAfdv971IU+kcPUczi7NVBQ==",++ "license": "MIT",++ "engines": {++ "node": ">=0.10.0"++ }++ },++ "node_modules/postgres-date": {++ "version": "1.0.7",++ "resolved": "https://registry.npmjs.org/postgres-date/-/postgres-date-1.0.7.tgz",++ "integrity": "sha512-suDmjLVQg78nMK2UZ454hAG+OAW+HQPZ6n++TNDUX+L0+uUlLywnoxJKDou51Zm+zTCjrCl0Nq6J9C5hP9vK/Q==",++ "license": "MIT",++ "engines": {++ "node": ">=0.10.0"++ }++ },++ "node_modules/postgres-interval": {++ "version": "1.2.0",++ "resolved": "https://registry.npmjs.org/postgres-interval/-/postgres-interval-1.2.0.tgz",++ "integrity": "sha512-9ZhXKM/rw350N1ovuWHbGxnGh/SNJ4cnxHiM0rxE4VN41wsg8P8zWn9hv/buK00RP4WvlOyr/RBDiptyxVbkZQ==",++ "license": "MIT",++ "dependencies": {++ "xtend": "^4.0.0"++ },++ "engines": {++ "node": ">=0.10.0"++ }++ },"node_modules/prebuild-install": {"version": "7.1.3","resolved": "https://registry.npmjs.org/prebuild-install/-/prebuild-install-7.1.3.tgz",@@@ -1286,6 +1525,27 @@"node": ">= 6"}},++ "node_modules/redis-errors": {++ "version": "1.2.0",++ "resolved": "https://registry.npmjs.org/redis-errors/-/redis-errors-1.2.0.tgz",++ "integrity": "sha512-1qny3OExCf0UvUV/5wpYKf2YwPcOqXzkwKKSmKHiE6ZMQs5heeE/c8eXK+PNllPvmjgAbfnsbpkGZWy8cBpn9w==",++ "license": "MIT",++ "engines": {++ "node": ">=4"++ }++ },++ "node_modules/redis-parser": {++ "version": "3.0.0",++ "resolved": "https://registry.npmjs.org/redis-parser/-/redis-parser-3.0.0.tgz",++ "integrity": "sha512-DJnGAeenTdpMEH6uAJRK/uiyEIH9WVsUmoLwzudwGJUwZPp80PDBWPHXSAGNPwNvIXAbe7MSUB1zQFugFml66A==",++ "license": "MIT",++ "dependencies": {++ "redis-errors": "^1.0.0"++ },++ "engines": {++ "node": ">=4"++ }++ },"node_modules/require-from-string": {"version": "2.0.2","resolved": "https://registry.npmjs.org/require-from-string/-/require-from-string-2.0.2.tgz",@@@ -1501,6 +1761,21 @@"simple-concat": "^1.0.0"}},++ "node_modules/split2": {++ "version": "4.2.0",++ "resolved": "https://registry.npmjs.org/split2/-/split2-4.2.0.tgz",++ "integrity": "sha512-UcjcJOWknrNkF6PLX83qcHM6KHgVKNkV62Y8a5uYDVv9ydGQVwAHMKqHdJje1VTWpljG0WYpCDhrCdAOYH4TWg==",++ "license": "ISC",++ "engines": {++ "node": ">= 10.x"++ }++ },++ "node_modules/standard-as-callback": {++ "version": "2.1.0",++ "resolved": "https://registry.npmjs.org/standard-as-callback/-/standard-as-callback-2.1.0.tgz",++ "integrity": "sha512-qoRRSyROncaz1z0mvYqIE4lCd9p2R90i6GxW3uZv5ucSu8tU7B5HXUP1gG8pVZsYNVaXjk8ClXHPttLyxAL48A==",++ "license": "MIT"++ },"node_modules/statuses": {"version": "2.0.2","resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.2.tgz",@@@ -1670,6 +1945,15 @@"optional": true}}++ },++ "node_modules/xtend": {++ "version": "4.0.2",++ "resolved": "https://registry.npmjs.org/xtend/-/xtend-4.0.2.tgz",++ "integrity": "sha512-LKYU1iAXJXUgAXn9URjiu+MWhyUXHsvfp7mcuYm9dSUKK0/CjtrUwFAxD82/mCWbtLsGjFIad0wIsod4zrTAEQ==",++ "license": "MIT",++ "engines": {++ "node": ">=0.4"++ }}}}@@@ -11,9 +11,16 @@"rtmp-ingest": "node workers/rtmp-ingest.js","restream-worker": "node workers/restream-worker.js","migrate-from-live": "node scripts/migrate-from-live.js",-- "test": "node test/run.js"++ "test": "node test/run.js",++ "test:pg": "OPENRE_TEST_STORE=pg node test/run.js"},-- "keywords": ["openvibe", "openre", "rtmp", "restream", "ingest"],++ "keywords": [++ "openvibe",++ "openre",++ "rtmp",++ "restream",++ "ingest"++ ],"author": "OpenVibers","license": "AGPL-3.0-only","repository": "github:OpenVibers/OpenRe.Stream",@@@ -22,12 +29,17 @@"cookie-parser": "^1.4.7","dotenv": "^16.4.7","express": "^4.21.2",++ "iovalkey": "^0.4.0","node-media-server": "2.7.4",-- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.49.0",-- "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.12.0",-- "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.25.0"++ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.76.0",++ "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.21.3",++ "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.28.0",++ "pg": "^8.23.0"},"engines": {"node": ">=22.12.0"++ },++ "devDependencies": {++ "@electric-sql/pglite": "^0.5.8"}}@@@ -15,7 +15,8 @@* bind OPENRE_RTMP_BIND is public and the RTMP port really listens on a public address* dns the ingest host resolves (A/AAAA) only to this host's addresses* port the ingest port answers an RTMP handshake from outside (see "port" below)-- * db PRAGMA integrity_check and foreign_key_check of openre.db (opened read-only)++ * db OpenRe's PostgreSQL database (DATABASE_URL of openre.env, one READ ONLY transaction): no data++ * checksum failure, no foreign key left unvalidated, the main tables' counts* live-env /etc/openvibe/live.env has OPENRE_URL, OPENRE_EVENTS_SECRET (32+ chars) and* OV_OAUTH_CLIENT_SECRET, and the running Live process was started with them* events Events has Live's openre.session.* subscription to /internal/openre-events, enabled,@@@ -27,7 +28,10 @@* that is not the host (a checkout with `npm ci` is enough), or pass --probe-url with a probe you* trust (GET, `{host}` and `{port}` replaced, JSON answer with a boolean `open` or `reachable`).*-- * Nothing is written anywhere: files and databases are opened read-only, no secret value is++ * OpenRe and Events serve from PostgreSQL (ADR-035): their databases are read through the DATABASE_URL of their env++ * files (--openre-db / --events-db take env:<file>, a postgres:// URL or a SQLite file); Live's is still its SQLite file.++ *++ * Nothing is written anywhere: files are opened read-only and PostgreSQL is read in READ ONLY transactions, no secret value is* printed, the only network traffic is DNS, one GET to /api/ready and one TCP connection with an* RTMP handshake (no publish). Exit 0 when no check FAILs (--strict: also no WARN/MANUAL).*/@@@ -43,8 +47,8 @@ const DEFAULTS = Object.freeze({root: '/opt/openre.stream',openreEnv: '/etc/openvibe/openre.env',liveEnv: '/etc/openvibe/live.env',-- openreDb: '/var/lib/openre/openre.db',-- eventsDb: '/var/lib/openvibe-events/events.db',++ openreDb: 'env:/etc/openvibe/openre.env',++ eventsDb: 'env:/etc/openvibe/events.env',liveDb: '/opt/openvibe.live/data/live.db',api: 'http://127.0.0.1:4500',openreUrl: 'http://127.0.0.1:4500',@@@ -134,17 +138,44 @@ function realDeps() {interfaces: () => os.networkInterfaces(),probe: rtmpProbe,fetch: (url, opts) => fetch(url, opts),-- openDb: (p) => {++ openDb: (target) => {++ if (/^postgres(ql)?:\/\//.test(target)) {++ const silent = { log() {}, info() {}, warn() {}, error() {} };++ return require('openvibe-sdk/db').createDb({ url: target, service: 'openre-preflight', max: 1, log: silent });++ }const Database = require('better-sqlite3');-- return new Database(p, { readonly: true, fileMustExist: true });++ return new Database(target, { readonly: true, fileMustExist: true });},};}--function withDb(deps, file, fn) {++/**++ * fn(db, kind) on a read-only handle. `target`: env:<file> (that file's DATABASE_URL, never printed), a postgres:// URL++ * or a SQLite file. PostgreSQL is read in one READ ONLY transaction (a write fails); SQLite is opened read-only.++ */++async function withDb(deps, target, fn) {++ let where = String(target);let db;-- try { db = deps.openDb(file); } catch (err) { return { error: `cannot open ${file} read-only: ${err.message}` }; }-- try { return fn(db); } finally { try { db.close(); } catch { /* read-only */ } }++ try {++ let url = where;++ if (where.startsWith('env:')) {++ const file = where.slice(4);++ where = `the DATABASE_URL of ${file}`;++ const env = parseEnvFile(deps.readFile(file));++ if (!env) return { error: `cannot read ${file}` };++ if (!env.DATABASE_URL) return { error: `${file} has no DATABASE_URL` };++ url = env.DATABASE_URL;++ } else if (/^postgres(ql)?:\/\//.test(where)) where = 'the PostgreSQL URL given';++ db = await deps.openDb(url);++ } catch (err) { return { error: `cannot open ${where} read-only: ${err.message}` }; }++ try {++ if (typeof db.tx === 'function') {++ return await db.tx(async () => { await db.exec('SET TRANSACTION READ ONLY'); return await fn(db, 'postgresql'); });++ }++ return await fn(db, 'sqlite');++ } catch (err) {++ return { error: `${where}: ${err.message}` };++ } finally { try { await db.close(); } catch { /* read-only */ } }}// ── checks ──────────────────────────────────────────────────────────────────@@@ -191,25 +222,25 @@ async function checkService(ctx) {if (res.status !== 200 || body.status !== 'ready') problems.push(`status ${res.status} ${body.status} (${JSON.stringify(body.checks || {})})`);if (body.mode === 'drill') problems.push('the API runs in drill mode (OPENRE_DRILL)');const workers = Array.isArray(body.workers) ? body.workers : [];-- const rows = ctx.workerRows || workerRowsFromDb(ctx);++ const rows = ctx.workerRows || await workerRowsFromDb(ctx);for (const kind of WORKER_KINDS) {-- const ready = workers.filter((w) => w.kind === kind && w.state === 'ready');++ const ready = await workers.filter((w) => w.kind === kind && w.state === 'ready');if (ready.length !== 1) { problems.push(`${ready.length} ready ${kind} workers (want 1)`); continue; }const row = rows && rows.find((w) => w.kind === kind && w.generation === ready[0].generation);if (ctx.release && row && row.release && row.release !== ctx.release) problems.push(`${kind}#${ready[0].generation} runs release ${row.release}, current is ${ctx.release}: deploy.sh workers`);}-- const draining = workers.filter((w) => w.state === 'draining').map((w) => `${w.kind}#${w.generation}`);++ const draining = (await workers.filter((w) => w.state === 'draining')).map((w) => `${w.kind}#${w.generation}`);if (!body.coordinator || !body.coordinator.lease_valid) problems.push('no valid coordinator lease');if (!body.events || !body.events.configured) problems.push('event relay not configured (EVENTS_URL, OV_OAUTH_CLIENT_SECRET)');if (problems.length) return ['FAIL', problems.join('; ')];-- const summary = WORKER_KINDS.map((k) => { const w = workers.find((x) => x.kind === k && x.state === 'ready'); return `${k}#${w.generation}`; }).join(', ');++ const summary = (await Promise.all(WORKER_KINDS.map(async (k) => { const w = await workers.find((x) => x.kind === k && x.state === 'ready'); return `${k}#${w.generation}`; }))).join(', ');const pending = body.events && body.events.pending ? `, ${body.events.pending} events waiting in the outbox` : '';if (draining.length) return ['WARN', `ready: ${summary}; still draining: ${draining.join(', ')}${pending}`];return ['PASS', `ready: ${summary}, coordinator lease valid, event relay on${pending}`];}--function workerRowsFromDb(ctx) {-- const r = withDb(ctx.deps, ctx.opts.openreDb, (db) => db.prepare("SELECT kind, generation, release, state FROM workers WHERE state IN ('starting', 'ready', 'draining')").all());++async function workerRowsFromDb(ctx) {++ const r = await withDb(ctx.deps, ctx.opts.openreDb, async (db) => await db.prepare("SELECT kind, generation, release, state FROM workers WHERE state IN ('starting', 'ready', 'draining')").all());ctx.workerRows = Array.isArray(r) ? r : null;return ctx.workerRows;}@@@ -312,20 +343,30 @@ async function checkPort(ctx) {}async function checkDb(ctx) {-- const r = withDb(ctx.deps, ctx.opts.openreDb, (db) => {-- const integrity = db.prepare('PRAGMA integrity_check').all().map((x) => Object.values(x)[0]).join('; ');-- const fk = db.prepare('PRAGMA foreign_key_check').all();++ const r = await withDb(ctx.deps, ctx.opts.openreDb, async (db, kind) => {++ let integrity = 'ok';++ let fk = 0;++ if (kind === 'sqlite') {++ integrity = (await db.prepare('PRAGMA integrity_check').all()).map((x) => Object.values(x)[0]).join('; ');++ fk = (await db.prepare('PRAGMA foreign_key_check').all()).length;++ } else {++ // PostgreSQL checks every key on write: what can still be wrong is a data page (checksums) or a key added NOT VALID.++ const c = await db.prepare('SELECT checksum_failures AS n FROM pg_stat_database WHERE datname = current_database()').get();++ if (c && Number(c.n) > 0) integrity = `${c.n} data checksum failure(s)`;++ fk = Number((await db.prepare("SELECT count(*) AS n FROM pg_constraint WHERE contype = 'f' AND NOT convalidated").get()).n);++ }const counts = {};for (const t of ['stream_definitions', 'ingest_keys', 'ingest_sessions', 'destinations', 'migration_map']) {-- try { counts[t] = db.prepare(`SELECT count(*) AS n FROM ${t}`).get().n; } catch { counts[t] = null; }++ try { counts[t] = (await db.prepare(`SELECT count(*) AS n FROM ${t}`).get()).n; } catch { counts[t] = null; }}-- return { integrity, fk: fk.length, counts };++ return { integrity, fk, kind, counts };});if (r.error) return ['FAIL', r.error];const counts = Object.entries(r.counts).map(([k, v]) => `${k} ${v == null ? '?' : v}`).join(', ');-- if (r.integrity !== 'ok') return ['FAIL', `integrity_check = ${r.integrity.slice(0, 300)}`];-- if (r.fk) return ['FAIL', `integrity_check ok, but ${r.fk} foreign key violation(s)`];-- return ['PASS', `integrity_check ok, no foreign key violations (${counts})`];++ const what = r.kind === 'sqlite' ? 'integrity_check' : 'PostgreSQL data checksums';++ if (r.integrity !== 'ok') return ['FAIL', `${what} = ${r.integrity.slice(0, 300)}`];++ if (r.fk) return ['FAIL', `${what} ok, but ${r.fk} foreign key ${r.kind === 'sqlite' ? 'violation(s)' : 'constraint(s) not validated'}`];++ return ['PASS', `${what} ok, no foreign key ${r.kind === 'sqlite' ? 'violations' : 'left unvalidated'} (${counts})`];}async function checkLiveEnv(ctx) {@@@ -360,10 +401,10 @@ async function checkLiveEnv(ctx) {async function checkEvents(ctx) {const { opts } = ctx;const env = liveEnv(ctx) || {};-- const r = withDb(ctx.deps, opts.eventsDb, (db) => {-- const subs = db.prepare("SELECT id, enabled, secret FROM subscriptions WHERE consumer = 'live' AND topic_pattern = ? AND endpoint = ?").all(opts.topic, opts.endpoint);-- const dead = subs.length ? db.prepare("SELECT count(*) AS n FROM deliveries WHERE subscription_id = ? AND status = 'dead'").get(subs[0].id).n : 0;-- const pending = subs.length ? db.prepare("SELECT count(*) AS n FROM deliveries WHERE subscription_id = ? AND status IN ('pending', 'failed')").get(subs[0].id).n : 0;++ const r = await withDb(ctx.deps, opts.eventsDb, async (db) => {++ const subs = await db.prepare("SELECT id, enabled, secret FROM subscriptions WHERE consumer = 'live' AND topic_pattern = ? AND endpoint = ?").all(opts.topic, opts.endpoint);++ const dead = subs.length ? (await db.prepare("SELECT count(*) AS n FROM deliveries WHERE subscription_id = ? AND status = 'dead'").get(subs[0].id)).n : 0;++ const pending = subs.length ? (await db.prepare("SELECT count(*) AS n FROM deliveries WHERE subscription_id = ? AND status IN ('pending', 'failed')").get(subs[0].id)).n : 0;return { subs, dead, pending };});if (r.error) return ['FAIL', r.error];@@@ -382,22 +423,22 @@ async function checkEvents(ctx) {async function checkSlot(ctx) {const { opts } = ctx;const id = Number(opts.slot);-- const live = withDb(ctx.deps, opts.liveDb, (db) => {-- const slot = db.prepare('SELECT id, user_id, slug, title, protocol, streaming_method, ingest_authority, openre_stream_id FROM managed_streams WHERE id = ?').get(id);++ const live = await withDb(ctx.deps, opts.liveDb, async (db) => {++ const slot = await db.prepare('SELECT id, user_id, slug, title, protocol, streaming_method, ingest_authority, openre_stream_id FROM managed_streams WHERE id = ?').get(id);if (!slot) return { slot: null };-- const liveNow = db.prepare('SELECT id FROM streams WHERE managed_stream_id = ? AND is_live = 1 LIMIT 1').get(id);-- const subject = db.prepare("SELECT subject_id FROM linked_accounts WHERE service = 'network' AND user_id = ?").get(slot.user_id);-- const user = db.prepare('SELECT username, is_banned FROM users WHERE id = ?').get(slot.user_id);++ const liveNow = await db.prepare('SELECT id FROM streams WHERE managed_stream_id = ? AND is_live = 1 LIMIT 1').get(id);++ const subject = await db.prepare("SELECT subject_id FROM linked_accounts WHERE service = 'network' AND user_id = ?").get(slot.user_id);++ const user = await db.prepare('SELECT username, is_banned FROM users WHERE id = ?').get(slot.user_id);return { slot, liveNow, subject: subject && subject.subject_id, user };});if (live.error) return ['FAIL', live.error];if (!live.slot) return ['FAIL', `Live has no managed stream ${id}`];const s = live.slot;-- const openre = withDb(ctx.deps, opts.openreDb, (db) => {-- const defs = db.prepare("SELECT d.id, d.state, d.mirror_to_live FROM external_refs r JOIN stream_definitions d ON d.id = r.definition_id WHERE r.service = 'live' AND r.type = 'managed_stream' AND r.ref_id = ?").all(String(id));-- const map = db.prepare("SELECT source_type, status, reason FROM migration_map WHERE source_system = 'live' AND ((source_type = 'managed_stream' AND source_id = ?) OR source_type = 'restream_destination')").all(String(id));-- const dests = defs.length ? db.prepare('SELECT enabled, hold_reason FROM destinations WHERE definition_id = ?').all(defs[0].id) : [];-- const openSessions = defs.length ? db.prepare("SELECT count(*) AS n FROM ingest_sessions WHERE definition_id = ? AND state IN ('starting', 'live', 'ending')").get(defs[0].id).n : 0;++ const openre = await withDb(ctx.deps, opts.openreDb, async (db) => {++ const defs = await db.prepare("SELECT d.id, d.state, d.mirror_to_live FROM external_refs r JOIN stream_definitions d ON d.id = r.definition_id WHERE r.service = 'live' AND r.type = 'managed_stream' AND r.ref_id = ?").all(String(id));++ const map = await db.prepare("SELECT source_type, status, reason FROM migration_map WHERE source_system = 'live' AND ((source_type = 'managed_stream' AND source_id = ?) OR source_type = 'restream_destination')").all(String(id));++ const dests = defs.length ? await db.prepare('SELECT enabled, hold_reason FROM destinations WHERE definition_id = ?').all(defs[0].id) : [];++ const openSessions = defs.length ? (await db.prepare("SELECT count(*) AS n FROM ingest_sessions WHERE definition_id = ? AND state IN ('starting', 'live', 'ending')").get(defs[0].id)).n : 0;return { defs, slotMap: map.find((m) => m.source_type === 'managed_stream') || null, dests, openSessions };});if (openre.error) return ['FAIL', openre.error];@@@ -481,4 +522,4 @@ if (require.main === module) {}, (err) => { console.error(`cutover-preflight: ${err.stack || err}`); process.exit(2); });}--module.exports = { preflight, parseArgs, format, parseEnvFile, parseSsListeners, localAddresses, rtmpProbe, sameSecret, CHECKS, DEFAULTS };++module.exports = { preflight, parseArgs, format, parseEnvFile, parseSsListeners, localAddresses, rtmpProbe, sameSecret, withDb, CHECKS, DEFAULTS };@@@ -24,7 +24,7 @@ function arg(name) {return i > 0 ? process.argv[i + 1] : null;}--function main() {++async function main() {require('dotenv').config({ path: process.env.OPENRE_ENV_FILE || path.join(process.cwd(), '.env') });const { load } = require('../server/config');const { openRuntime } = require('../server/store');@@@ -35,9 +35,9 @@ function main() {const config = load();if (apply && !config.secretsKey) { console.error('OPENRE_SECRETS_KEY is required to import destinations'); process.exit(2); }const liveDb = new Database(liveDbPath, { readonly: true, fileMustExist: true });-- const rt = openRuntime({ config, log: { log() {}, warn: console.warn, error: console.error } });++ const rt = await openRuntime({ config, log: { log() {}, warn: console.warn, error: console.error } });const onlySlots = arg('slots') ? arg('slots').split(',').map(Number).filter(Number.isFinite) : null;-- const report = migrate({ liveDb, rt, apply, onlySlots });++ const report = await migrate({ liveDb, rt, apply, onlySlots });const r = config.rtmp;const md = checklist(report, { openreUrl: config.baseUrl, rtmpUrl: `rtmp://${r.publicHost}${r.publicPort === 1935 ? '' : `:${r.publicPort}`}/live` });const out = arg('checklist');nnew file mode 100755@@@ -0,0 +1,27 @@++#!/usr/bin/env node++'use strict';++/**++ * The one-time move of OpenRe's SQLite database (OPENRE_DB_PATH) into its PostgreSQL schema (ADR-035; the procedure is++ * openvibe-sdk docs/migrating-to-postgresql.md, section 6, carried out by openvibe-sdk/db runSqliteMigration).++ *++ * node scripts/migrate-to-postgres.js [--sqlite <file>] [--pglite] [--json]++ *++ * Applies migrations/ as the owner (DATABASE_DIRECT_URL), copies every table into emptied tables, verifies row counts++ * and checksums, and exits 1 unless everything verified. The SQLite file is opened read-only. Every table keeps its++ * name and columns; the SDK outbox's rows move into its PostgreSQL outbox (the same columns). Run with every OpenRe++ * process stopped (OpenVibe.Host roles/data/switch-service.sh with SWITCH_UNITS).++ */++require('dotenv').config({ path: process.env.OPENRE_ENV_FILE || require('path').join(process.cwd(), '.env') });++const { runSqliteMigration } = require('openvibe-sdk/db');++const { load } = require('../server/config');++const { MIGRATIONS } = require('../server/db');++++const TABLES = {};++++if (require.main === module) {++ const config = load();++ runSqliteMigration({ service: 'openre', sqlite: config.dbPath, directUrl: config.db.directUrl, migrations: MIGRATIONS, tables: TABLES })++ .then((code) => process.exit(code), (err) => { console.error(`migrate-to-postgres failed: ${err.message}`); process.exit(1); });++}++++module.exports = { TABLES };@@@ -10,7 +10,7 @@*/'use strict';--const { createActorLimiter } = require('openvibe-sdk/limits');++const { createActorLimiter, createValkeyLimitStore } = require('openvibe-sdk/limits');const num = (v, d) => { const n = parseInt(v, 10); return Number.isFinite(n) && n > 0 ? n : d; };@@@ -23,7 +23,7 @@ const ROUTES = [['openre.stream.create', /^POST$/, /^\/streams\/?$/, { minute: 10, hour: 100 }],];--function createOpenReActorLimits({ env = process.env, registry = null, now } = {}) {++function createOpenReActorLimits({ env = process.env, registry = null, now, valkey = null } = {}) {const refused = registry && typeof registry.counter === 'function'? registry.counter({ name: 'openre_rate_limited_total', help: 'API writes refused 429 by a per-actor limit, by limit name and window', labelNames: ['limit', 'window'] }): null;@@@ -31,6 +31,8 @@ function createOpenReActorLimits({ env = process.env, registry = null, now } = {limits: { minute: num(env.OPENRE_LIMITS_MINUTE, 120), hour: num(env.OPENRE_LIMITS_HOUR, 3000) },actor: (req) => (req.caller && req.caller.kind === 'user' && req.caller.subject ? `user:${req.caller.subject}` : null),...(now ? { now } : {}),++ // Valkey (ADR-035): counters survive a restart of openre-api and are shared if it ever runs twice.++ ...(valkey ? { store: createValkeyLimitStore(valkey) } : {}),onLimited(e) {console.warn(`[Limits] ${e.name}: ${e.actor} refused, over ${e.limit} per ${e.window}`);if (refused) refused.inc({ limit: e.name, window: e.window });@@@ -46,9 +46,9 @@ function createV1Router({ rt, auth }) {}};-- function publicDefinition(d, { withSessions = true } = {}) {-- const keys = store.definitions.keys(d.id).filter(k => k.status !== 'revoked');-- const open = withSessions ? store.sessions.openFor(d.id)[0] || null : null;++ async function publicDefinition(d, { withSessions = true } = {}) {++ const keys = (await store.definitions.keys(d.id)).filter(k => k.status !== 'revoked');++ const open = withSessions ? (await store.sessions.openFor(d.id))[0] || null : null;return {id: d.id,title: d.title,@@@ -62,7 +62,7 @@ function createV1Router({ rt, auth }) {state: d.state,revision: d.revision,external_refs: d.external_refs,-- ingest: store.definitions.ingestEndpoints(d),++ ingest: await store.definitions.ingestEndpoints(d),keys: keys.map(k => ({ id: k.id, hint: k.hint, status: k.status, grace_until: iso(k.grace_until), created_at: iso(k.created_at), last_used_at: iso(k.last_used_at) })),session: open ? { id: open.id, state: open.state, protocol: open.protocol, live_at: iso(open.live_at) } : null,created_at: iso(d.created_at),@@@ -70,7 +70,7 @@ function createV1Router({ rt, auth }) {};}-- function publicSession(s, { detail = false } = {}) {++ async function publicSession(s, { detail = false } = {}) {const out = {id: s.id,stream_id: s.definition_id,@@@ -90,17 +90,17 @@ function createV1Router({ rt, auth }) {duration_seconds: s.live_at ? Math.round(((s.ended_at || Date.now()) - s.live_at) / 1000) : 0,};if (detail) {-- out.transitions = store.sessions.transitions(s.id).map(t => ({ ...t, at: iso(t.at) }));-- out.outputs = store.outputs.outputsOfSession(s.id);-- const rec = store.recordings.bySession(s.id);++ out.transitions = (await store.sessions.transitions(s.id)).map(t => ({ ...t, at: iso(t.at) }));++ out.outputs = await store.outputs.outputsOfSession(s.id);++ const rec = await store.recordings.bySession(s.id);out.recording = rec ? { id: rec.id, mode: rec.mode, state: rec.state, media: rec.media_vod_id ? { app: rec.media_app, vod_id: rec.media_vod_id } : null, last_error: rec.last_error } : null;-- out.playback = store.sessions.playback(s);++ out.playback = await store.sessions.playback(s);}return out;}-- function loadDefinition(req, res, id) {-- const d = store.definitions.get(id);++ async function loadDefinition(req, res, id) {++ const d = await store.definitions.get(id);if (!d || d.state === 'archived' || !auth.canAccess(req.caller, d.owner_subject)) {fail(req, res, 404, 'openre.stream_not_found', 'no such stream definition');return null;@@@ -108,9 +108,9 @@ function createV1Router({ rt, auth }) {return d;}-- function loadDestination(req, res, id) {-- const dest = store.outputs.destinationRow(id);-- const d = dest ? store.definitions.get(dest.definition_id) : null;++ async function loadDestination(req, res, id) {++ const dest = await store.outputs.destinationRow(id);++ const d = dest ? await store.definitions.get(dest.definition_id) : null;if (!dest || !d || !auth.canAccess(req.caller, d.owner_subject)) {fail(req, res, 404, 'openre.destination_not_found', 'no such destination');return null;@@@ -118,9 +118,9 @@ function createV1Router({ rt, auth }) {return dest;}-- function loadSession(req, res, id) {-- const s = store.sessions.get(id);-- const d = s ? store.definitions.row(s.definition_id) : null;++ async function loadSession(req, res, id) {++ const s = await store.sessions.get(id);++ const d = s ? await store.definitions.row(s.definition_id) : null;if (!s || !d || !auth.canAccess(req.caller, d.owner_subject)) {fail(req, res, 404, 'openre.session_not_found', 'no such session');return null;@@@ -130,29 +130,29 @@ function createV1Router({ rt, auth }) {// ── Streams ───────────────────────────────────────────────-- router.get('/streams', guard('openre.stream.read'), handle((req, res) => {++ router.get('/streams', guard('openre.stream.read'), handle(async (req, res) => {const ref = String(req.query.external_ref || '');if (ref) {const [service, type, ...rest] = ref.split(':');-- const d = store.definitions.findByRef(service, type, rest.join(':'));++ const d = await store.definitions.findByRef(service, type, rest.join(':'));const visible = d && d.state !== 'archived' && auth.canAccess(req.caller, d.owner_subject);-- return res.json({ streams: visible ? [publicDefinition(d)] : [] });++ return res.json({ streams: visible ? [await publicDefinition(d)] : [] });}const c = req.caller;let owner = c.subject || null;if (c.kind === 'user' && c.staff && req.query.all === '1') owner = null;-- const list = store.definitions.list({ owner_subject: owner || undefined, limit: Number(req.query.limit) || 100 });-- return res.json({ streams: list.map(d => publicDefinition(d)) });++ const list = await store.definitions.list({ owner_subject: owner || undefined, limit: Number(req.query.limit) || 100 });++ return res.json({ streams: (await Promise.all(list.map(async d => await publicDefinition(d)))) });}));-- router.post('/streams', guard('openre.stream.write'), handle((req, res) => {++ router.post('/streams', guard('openre.stream.write'), handle(async (req, res) => {const owner = req.caller.kind === 'user' ? req.caller.subject : (req.caller.subject || (req.body && req.body.owner_subject));if (!owner) return fail(req, res, 400, 'openre.invalid_owner', 'name the owner (X-OV-Subject)');if (req.caller.kind === 'service' && req.caller.subject && req.body && req.body.owner_subject && req.body.owner_subject !== req.caller.subject) {return fail(req, res, 400, 'openre.invalid_owner', 'owner_subject differs from X-OV-Subject');}const b = req.body || {};-- const { definition, key } = store.definitions.create({++ const { definition, key } = await store.definitions.create({owner_subject: owner,title: b.title, description: b.description, protocols: b.protocols,recording_mode: b.recording_mode, recording_visibility: b.recording_visibility, playback_visibility: b.playback_visibility,@@@ -160,45 +160,45 @@ function createV1Router({ rt, auth }) {created_by: auth.actorOf(req.caller),});res.set('Cache-Control', 'no-store');-- return res.status(201).json({ stream: publicDefinition(definition), key: { id: key.id, key: key.key, hint: key.hint, shown_once: true } });++ return res.status(201).json({ stream: await publicDefinition(definition), key: { id: key.id, key: key.key, hint: key.hint, shown_once: true } });}));-- router.get('/streams/:id', guard('openre.stream.read'), handle((req, res) => {-- const d = loadDefinition(req, res, req.params.id);-- if (d) res.json({ stream: publicDefinition(d) });++ router.get('/streams/:id', guard('openre.stream.read'), handle(async (req, res) => {++ const d = await loadDefinition(req, res, req.params.id);++ if (d) res.json({ stream: await publicDefinition(d) });}));-- router.patch('/streams/:id', guard('openre.stream.write'), handle((req, res) => {-- const d = loadDefinition(req, res, req.params.id);++ router.patch('/streams/:id', guard('openre.stream.write'), handle(async (req, res) => {++ const d = await loadDefinition(req, res, req.params.id);if (!d) return;const b = req.body || {};-- const updated = store.definitions.update(d.id, {++ const updated = await store.definitions.update(d.id, {title: b.title, description: b.description, protocols: b.protocols, recording_mode: b.recording_mode,recording_visibility: b.recording_visibility, playback_visibility: b.playback_visibility, mirror_to_live: b.mirror_to_live, state: b.state,});-- res.json({ stream: publicDefinition(updated) });++ res.json({ stream: await publicDefinition(updated) });}));-- router.delete('/streams/:id', guard('openre.stream.write'), handle((req, res) => {-- const d = loadDefinition(req, res, req.params.id);++ router.delete('/streams/:id', guard('openre.stream.write'), handle(async (req, res) => {++ const d = await loadDefinition(req, res, req.params.id);if (!d) return;-- store.definitions.archive(d.id);++ await store.definitions.archive(d.id);res.status(204).end();}));-- router.get('/streams/:id/keys', guard('openre.stream.read'), handle((req, res) => {-- const d = loadDefinition(req, res, req.params.id);-- if (d) res.json({ keys: store.definitions.keys(d.id).map(k => ({ ...k, created_at: iso(k.created_at), grace_until: iso(k.grace_until), revoked_at: iso(k.revoked_at), last_used_at: iso(k.last_used_at) })) });++ router.get('/streams/:id/keys', guard('openre.stream.read'), handle(async (req, res) => {++ const d = await loadDefinition(req, res, req.params.id);++ if (d) res.json({ keys: (await store.definitions.keys(d.id)).map(k => ({ ...k, created_at: iso(k.created_at), grace_until: iso(k.grace_until), revoked_at: iso(k.revoked_at), last_used_at: iso(k.last_used_at) })) });}));-- router.post('/streams/:id/keys/rotate', guard('openre.key.rotate'), handle((req, res) => {-- const d = loadDefinition(req, res, req.params.id);++ router.post('/streams/:id/keys/rotate', guard('openre.key.rotate'), handle(async (req, res) => {++ const d = await loadDefinition(req, res, req.params.id);if (!d) return;const b = req.body || {};-- const r = store.definitions.rotateKey(d.id, { grace_seconds: b.grace_seconds, rotated_by: auth.actorOf(req.caller), reason: 'rotated' });++ const r = await store.definitions.rotateKey(d.id, { grace_seconds: b.grace_seconds, rotated_by: auth.actorOf(req.caller), reason: 'rotated' });let ended = 0;if (b.end_sessions) {-- for (const s of store.sessions.openFor(d.id)) if (store.sessions.requestEnd(s.id, `key_rotation:${auth.actorOf(req.caller)}`)) ended++;++ for (const s of await store.sessions.openFor(d.id)) if (await store.sessions.requestEnd(s.id, `key_rotation:${auth.actorOf(req.caller)}`)) ended++;}res.set('Cache-Control', 'no-store');res.json({@@@ -206,63 +206,63 @@ function createV1Router({ rt, auth }) {retired: r.retired,grace_until: iso(r.grace_until),sessions_ending: ended,-- ingest: store.definitions.ingestEndpoints(store.definitions.get(d.id)),++ ingest: await store.definitions.ingestEndpoints(await store.definitions.get(d.id)),});}));// ── Destinations ──────────────────────────────────────────-- router.get('/streams/:id/destinations', guard('openre.output.read'), handle((req, res) => {-- const d = loadDefinition(req, res, req.params.id);-- if (d) res.json({ destinations: store.outputs.destinations(d.id) });++ router.get('/streams/:id/destinations', guard('openre.output.read'), handle(async (req, res) => {++ const d = await loadDefinition(req, res, req.params.id);++ if (d) res.json({ destinations: await store.outputs.destinations(d.id) });}));-- router.post('/streams/:id/destinations', guard('openre.output.write'), handle((req, res) => {-- const d = loadDefinition(req, res, req.params.id);-- if (d) res.status(201).json({ destination: store.outputs.createDestination(d.id, req.body || {}) });++ router.post('/streams/:id/destinations', guard('openre.output.write'), handle(async (req, res) => {++ const d = await loadDefinition(req, res, req.params.id);++ if (d) res.status(201).json({ destination: await store.outputs.createDestination(d.id, req.body || {}) });}));-- router.patch('/destinations/:id', guard('openre.output.write'), handle((req, res) => {-- const dest = loadDestination(req, res, req.params.id);-- if (dest) res.json({ destination: store.outputs.updateDestination(dest.id, req.body || {}) });++ router.patch('/destinations/:id', guard('openre.output.write'), handle(async (req, res) => {++ const dest = await loadDestination(req, res, req.params.id);++ if (dest) res.json({ destination: await store.outputs.updateDestination(dest.id, req.body || {}) });}));-- router.delete('/destinations/:id', guard('openre.output.write'), handle((req, res) => {-- const dest = loadDestination(req, res, req.params.id);++ router.delete('/destinations/:id', guard('openre.output.write'), handle(async (req, res) => {++ const dest = await loadDestination(req, res, req.params.id);if (!dest) return;-- store.outputs.deleteDestination(dest.id);++ await store.outputs.deleteDestination(dest.id);res.status(204).end();}));router.post('/destinations/:id/test', guard('openre.output.write'), handle(async (req, res) => {-- const dest = loadDestination(req, res, req.params.id);++ const dest = await loadDestination(req, res, req.params.id);if (!dest) return;const result = await testDestination(dest, { allowPrivate: config.outputs.allowPrivateHosts });-- store.outputs.log(null, dest.id, result.ok ? 'info' : 'warn', `test ${result.ok ? 'passed' : 'failed'}: ${result.checks.map(c => `${c.check} ${c.ok ? 'ok' : 'FAIL'} (${c.detail})`).join('; ')}`);++ await store.outputs.log(null, dest.id, result.ok ? 'info' : 'warn', `test ${result.ok ? 'passed' : 'failed'}: ${result.checks.map(c => `${c.check} ${c.ok ? 'ok' : 'FAIL'} (${c.detail})`).join('; ')}`);res.json(result);}));-- router.post('/destinations/:id/start', guard('openre.output.write'), handle((req, res) => {-- const dest = loadDestination(req, res, req.params.id);-- if (dest) res.json({ output: store.outputs.startDestination(dest.id) });++ router.post('/destinations/:id/start', guard('openre.output.write'), handle(async (req, res) => {++ const dest = await loadDestination(req, res, req.params.id);++ if (dest) res.json({ output: await store.outputs.startDestination(dest.id) });}));-- router.post('/destinations/:id/stop', guard('openre.output.write'), handle((req, res) => {-- const dest = loadDestination(req, res, req.params.id);-- if (dest) res.json({ stopping: store.outputs.stopDestination(dest.id) });++ router.post('/destinations/:id/stop', guard('openre.output.write'), handle(async (req, res) => {++ const dest = await loadDestination(req, res, req.params.id);++ if (dest) res.json({ stopping: await store.outputs.stopDestination(dest.id) });}));-- router.get('/destinations/:id/logs', guard('openre.output.read'), handle((req, res) => {-- const dest = loadDestination(req, res, req.params.id);-- if (dest) res.json({ logs: store.outputs.logsOfDestination(dest.id, Math.min(Number(req.query.limit) || 100, 500)).map(l => ({ ...l, at: iso(l.at) })) });++ router.get('/destinations/:id/logs', guard('openre.output.read'), handle(async (req, res) => {++ const dest = await loadDestination(req, res, req.params.id);++ if (dest) res.json({ logs: (await store.outputs.logsOfDestination(dest.id, Math.min(Number(req.query.limit) || 100, 500))).map(l => ({ ...l, at: iso(l.at) })) });}));// ── Sessions ──────────────────────────────────────────────-- router.get('/sessions', guard('openre.session.read'), handle((req, res) => {++ router.get('/sessions', guard('openre.session.read'), handle(async (req, res) => {const q = req.query;if (q.stream_id) {-- const d = loadDefinition(req, res, String(q.stream_id));++ const d = await loadDefinition(req, res, String(q.stream_id));if (!d) return;}const c = req.caller;@@@ -270,48 +270,48 @@ function createV1Router({ rt, auth }) {// sessions of the subject it acts for, or every session without X-OV-Subject.let owner = c.subject || null;if (c.kind === 'user' && c.staff && q.all === '1') owner = null;-- const list = store.sessions.list({++ const list = await store.sessions.list({definition_id: q.stream_id ? String(q.stream_id) : undefined,owner_subject: q.stream_id ? undefined : (owner || undefined),state: q.state ? String(q.state) : undefined,limit: q.limit,before: q.before,});-- res.json({ sessions: list.map(s => publicSession(s)) });++ res.json({ sessions: (await Promise.all(list.map(async s => await publicSession(s)))) });}));-- router.get('/sessions/:id', guard('openre.session.read'), handle((req, res) => {-- const s = loadSession(req, res, req.params.id);-- if (s) res.json({ session: publicSession(s, { detail: true }) });++ router.get('/sessions/:id', guard('openre.session.read'), handle(async (req, res) => {++ const s = await loadSession(req, res, req.params.id);++ if (s) res.json({ session: await publicSession(s, { detail: true }) });}));-- router.get('/sessions/:id/playback', guard('openre.session.read'), handle((req, res) => {-- const s = loadSession(req, res, req.params.id);-- if (s) res.json({ playback: store.sessions.playback(s) });++ router.get('/sessions/:id/playback', guard('openre.session.read'), handle(async (req, res) => {++ const s = await loadSession(req, res, req.params.id);++ if (s) res.json({ playback: await store.sessions.playback(s) });}));-- router.post('/sessions/:id/end', guard('openre.session.end'), handle((req, res) => {-- const s = loadSession(req, res, req.params.id);++ router.post('/sessions/:id/end', guard('openre.session.end'), handle(async (req, res) => {++ const s = await loadSession(req, res, req.params.id);if (!s) return;-- const requested = store.sessions.requestEnd(s.id, auth.actorOf(req.caller));-- res.status(requested ? 202 : 409).json({ requested, state: store.sessions.get(s.id).state });++ const requested = await store.sessions.requestEnd(s.id, auth.actorOf(req.caller));++ res.status(requested ? 202 : 409).json({ requested, state: (await store.sessions.get(s.id)).state });}));-- router.get('/sessions/:id/outputs', guard('openre.output.read'), handle((req, res) => {-- const s = loadSession(req, res, req.params.id);-- if (s) res.json({ outputs: store.outputs.outputsOfSession(s.id) });++ router.get('/sessions/:id/outputs', guard('openre.output.read'), handle(async (req, res) => {++ const s = await loadSession(req, res, req.params.id);++ if (s) res.json({ outputs: await store.outputs.outputsOfSession(s.id) });}));-- router.get('/outputs/:id/logs', guard('openre.output.read'), handle((req, res) => {-- const o = store.outputs.outputRow(req.params.id);-- const s = o ? loadSession(req, res, o.session_id) : null;++ router.get('/outputs/:id/logs', guard('openre.output.read'), handle(async (req, res) => {++ const o = await store.outputs.outputRow(req.params.id);++ const s = o ? await loadSession(req, res, o.session_id) : null;if (!o) return fail(req, res, 404, 'openre.output_not_found', 'no such output');-- if (s) res.json({ logs: store.outputs.logsOfOutput(o.id, Math.min(Number(req.query.limit) || 100, 500)).map(l => ({ ...l, at: iso(l.at) })) });++ if (s) res.json({ logs: (await store.outputs.logsOfOutput(o.id, Math.min(Number(req.query.limit) || 100, 500))).map(l => ({ ...l, at: iso(l.at) })) });}));-- router.get('/workers', guard('openre.session.read'), handle((req, res) => {++ router.get('/workers', guard('openre.session.read'), handle(async (req, res) => {if (req.caller.kind === 'user' && !req.caller.staff) return fail(req, res, 403, 'capability.denied', 'staff only');-- res.json({ workers: store.workers.recent(100).map(w => ({ ...w, started_at: iso(w.started_at), ready_at: iso(w.ready_at), heartbeat_at: iso(w.heartbeat_at), drain_started_at: iso(w.drain_started_at), drain_deadline: iso(w.drain_deadline), stopped_at: iso(w.stopped_at) })) });++ res.json({ workers: (await store.workers.recent(100)).map(w => ({ ...w, started_at: iso(w.started_at), ready_at: iso(w.ready_at), heartbeat_at: iso(w.heartbeat_at), drain_started_at: iso(w.drain_started_at), drain_deadline: iso(w.drain_deadline), stopped_at: iso(w.stopped_at) })) });}));return { router, publicDefinition, publicSession };@@@ -22,7 +22,7 @@ function createApp({ rt, auth, keys, log = console, fetchImpl }) {// sessions by state, from openvibe-shared/metrics (mounted before every route).const metrics = require('openvibe-shared/metrics').instrument(app, { service: 'openre' });metrics.registry.gauge({ name: 'openre_sessions', help: 'Ingest sessions by state', labelNames: ['state'],-- collect: () => db.prepare("SELECT state, count(*) AS n FROM ingest_sessions WHERE state IN ('starting','live') GROUP BY state").all().map((r) => ({ labels: { state: r.state }, value: r.n })) });++ collect: async () => (await db.prepare("SELECT state, count(*) AS n FROM ingest_sessions WHERE state IN ('starting','live') GROUP BY state").all()).map((r) => ({ labels: { state: r.state }, value: r.n })) });app.use(contracts.http.middleware());// One W3C trace across services (openvibe-shared/trace): calls made while serving a request carry its traceparent.require('openvibe-shared/trace').install(app);@@@ -64,46 +64,50 @@ function createApp({ rt, auth, keys, log = console, fetchImpl }) {// serving reads, rather than failing it. The worker list, the lease and the outbox stay in the body for// scripts/cutover-preflight.js; a restore drill never publishes, so its relay check is skipped, never ok.const { createReadiness, skip } = require('openvibe-shared/ready');-- const workerView = () => store.workers.alive().map(w => ({ kind: w.kind, generation: w.generation, state: w.state, heartbeat_age_ms: Date.now() - w.heartbeat_at }));-- const leaseView = () => {-- const l = db.prepare("SELECT holder, expires_at FROM leases WHERE name = 'coordinator'").get();++ const workerView = async () => (await store.workers.alive()).map(w => ({ kind: w.kind, generation: w.generation, state: w.state, heartbeat_age_ms: Date.now() - w.heartbeat_at }));++ const leaseView = async () => {++ const l = await db.prepare("SELECT holder, expires_at FROM leases WHERE name = 'coordinator'").get();return l ? { holder: l.holder, lease_valid: l.expires_at > Date.now() } : null;};++ // Valkey (ADR-035): the per-actor limit counters (optional: without VALKEY_URL they count in this process).++ const valkey = config.valkey.url ? require('openvibe-sdk/valkey').createValkey({ url: config.valkey.url, prefix: config.valkey.prefix }) : null;++ app.locals.valkey = valkey;const readiness = createReadiness({service: 'openre',release: release.release,checks: [-- { name: 'db', required: true, description: 'SQLite answers a query', check: () => db.prepare('SELECT 1 AS ok').get().ok === 1 },++ { name: 'db', required: true, description: 'SQLite answers a query', check: async () => (await db.prepare('SELECT 1 AS ok').get()).ok === 1 },{ name: 'network_key', required: true, description: 'OpenVibe.Network RS256 key (sign-in and service tokens)', check: () => keys.loaded() || 'Network public key not loaded yet' },{name: 'workers', required: false, description: 'a ready rtmp-ingest and restream worker',-- check: () => {-- const alive = workerView();++ check: async () => {++ const alive = await workerView();const missing = WORKER_KINDS.filter(k => !alive.some(w => w.kind === k && w.state === 'ready'));return missing.length ? `no ready ${missing.join(' or ')} worker` : { ok: true, detail: { ready: alive.filter(w => w.state === 'ready').map(w => `${w.kind}#${w.generation}`) } };},},-- { name: 'coordinator', required: false, description: 'the coordinator holds a valid lease', check: () => { const l = leaseView(); return (l && l.lease_valid) || 'no valid coordinator lease'; } },++ { name: 'valkey', required: false, description: 'per-actor limit counters', check: async () => (valkey ? await valkey.ready() : skip('VALKEY_URL not set: limits count in this process')) },++ { name: 'coordinator', required: false, description: 'the coordinator holds a valid lease', check: async () => { const l = await leaseView(); return (l && l.lease_valid) || 'no valid coordinator lease'; } },{name: 'events_relay', required: false, description: 'durable events to OpenVibe.Events',-- check: () => {-- const st = events.status();++ check: async () => {++ const st = await events.status();if (!st.configured) return config.drill ? skip('restore drill: never publishes') : 'relay off (EVENTS_URL or OV_OAUTH_CLIENT_SECRET unset): openre.* events wait in the outbox';return { ok: true, detail: { pending: st.pending, rejected: st.rejected } };},},],-- details: (body) => {++ details: async (body) => {let workers = [];let coordinator = null;-- try { workers = workerView(); coordinator = leaseView(); } catch { /* reported as empty */ }++ try { workers = await workerView(); coordinator = await leaseView(); } catch { /* reported as empty */ }const dbOk = body.checks.db && body.checks.db.status === 'ok';return {...(config.drill ? { mode: 'drill' } : {}),-- store: { engine: 'sqlite', path_configured: Boolean(config.dbPath) },++ store: { engine: 'postgresql' },workers,coordinator,-- events: dbOk ? events.status() : null,++ events: dbOk ? await events.status() : null,};},});@@@ -112,17 +116,17 @@ function createApp({ rt, auth, keys, log = console, fetchImpl }) {const apiJson = express.json({ limit: '256kb', type: ['application/json', 'application/*+json'] });const v1 = createV1Router({ rt, auth });// Per-actor limits on a signed-in person's writes (server/api/actor-limits.js; roadmap WS-R task 4).-- const actorLimits = require('./api/actor-limits').createOpenReActorLimits({ registry: metrics.registry });++ const actorLimits = require('./api/actor-limits').createOpenReActorLimits({ registry: metrics.registry, valkey });app.use('/api/v1', apiJson, auth.middleware({ services: true }), actorLimits, v1.router);// Public HTTP-FLV playback of a live session, proxied from the worker that holds it. A viewer// deploy of this API interrupts viewers of this URL (they reconnect), never the ingest.-- app.get('/play/:file', (req, res) => {++ app.get('/play/:file', async (req, res) => {const m = SESSION_FLV_RE.exec(req.params.file);-- const s = m ? store.sessions.get(m[1]) : null;-- const d = s ? store.definitions.row(s.definition_id) : null;++ const s = m ? await store.sessions.get(m[1]) : null;++ const d = s ? await store.definitions.row(s.definition_id) : null;if (!s || !d || s.state !== 'live' || d.playback_visibility === 'private') return res.status(404).end();-- const pb = store.sessions.playback(s);++ const pb = await store.sessions.playback(s);if (!pb || !pb.flv) return res.status(404).end();const upstream = http.get(pb.flv.internal_url, (up) => {if (up.statusCode !== 200) { res.status(502).end(); up.resume(); return; }@@@ -54,7 +54,7 @@ function createKeyStore({ urls = [], pem = null, fetchImpl = globalThis.fetch, lreturn null;}-- function start() {++ async function start() {if (pem) return Promise.resolve(key);const attempt = async () => {const k = await fetchOnce();@@@ -63,7 +63,7 @@ function createKeyStore({ urls = [], pem = null, fetchImpl = globalThis.fetch, l};refreshTimer = setInterval(() => { fetchOnce().catch(() => {}); }, 6 * 60 * 60 * 1000);refreshTimer.unref?.();-- return attempt();++ return await attempt();}return { get: () => key, loaded: () => Boolean(key), start, stop() { clearTimeout(retryTimer); clearInterval(refreshTimer); }, fetchOnce };@@@ -57,6 +57,11 @@ function load(env = process.env) {trustProxy: env.TRUST_PROXY != null ? Number(env.TRUST_PROXY) : 2,// ── Store ───────────────────────────────────────────────++ // PostgreSQL (ADR-035): DATABASE_URL serves through PgBouncer, DATABASE_DIRECT_URL (the owner) runs migrations/.++ // Without DATABASE_URL, development and tests use an embedded PGlite database (OPENRE_PGLITE_DIR). dbPath: the++ // SQLite file the one-time move reads (scripts/migrate-to-postgres.js).++ db: { url: env.DATABASE_URL || '', directUrl: env.DATABASE_DIRECT_URL || '', pgliteDir: env.OPENRE_PGLITE_DIR || '' },++ valkey: { url: env.VALKEY_URL || '', prefix: env.VALKEY_PREFIX || 'ov:openre:' },dbPath: env.OPENRE_DB_PATH || './data/openre.db',// ── Secrets ─────────────────────────────────────────────@@@ -33,59 +33,59 @@ function createCoordinator({ rt, media, lineage = null, log = console, holder =let ticking = null;const stats = { ticks: 0, lastTickAt: null, lastError: null, holdsLease: false };-- function takeLease() {-- const ok = lease.take.run({ holder, exp: now() + config.workers.leaseMs, now: now() }).changes > 0;++ async function takeLease() {++ const ok = (await lease.take.run({ holder, exp: now() + config.workers.leaseMs, now: now() })).changes > 0;stats.holdsLease = ok;return ok;}-- function failSessionsOf(worker, reason) {++ async function failSessionsOf(worker, reason) {let n = 0;-- for (const s of store.sessions.ofWorker(worker.id)) {-- const r = store.sessions.transition(s.id, 'failed', { reason, actor: 'coordinator' });++ for (const s of await store.sessions.ofWorker(worker.id)) {++ const r = await store.sessions.transition(s.id, 'failed', { reason, actor: 'coordinator' });if (r.ok) n++;}return n;}/** The synchronous part of a tick (everything but Media calls and the relay). */-- function syncTick() {++ async function syncTick() {const out = { lost: 0, failed: 0, drained: 0, outputsCreated: 0, outputsAssigned: 0, keysRevoked: 0, recordingsCreated: 0 };// 2. liveness-- for (const w of store.workers.stale(now() - config.workers.leaseMs)) {-- if (store.workers.lose(w.id, 'heartbeat_missed')) {++ for (const w of await store.workers.stale(now() - config.workers.leaseMs)) {++ if (await store.workers.lose(w.id, 'heartbeat_missed')) {out.lost++;-- out.failed += failSessionsOf(w, 'worker_lost');-- store.outputs.release(w.id);++ out.failed += await failSessionsOf(w, 'worker_lost');++ await store.outputs.release(w.id);log.warn(`[coordinator] worker ${w.kind}#${w.generation} (${w.id}) lost; sessions failed, outputs released`);}}// 3. orphans (lease expired, worker not alive)-- for (const s of store.sessions.expiredLeases()) {-- const w = s.worker_id ? store.workers.get(s.worker_id) : null;++ for (const s of await store.sessions.expiredLeases()) {++ const w = s.worker_id ? await store.workers.get(s.worker_id) : null;if (w && ['starting', 'ready', 'draining'].includes(w.state) && w.heartbeat_at >= now() - config.workers.leaseMs) continue;-- if (store.sessions.transition(s.id, 'failed', { reason: 'lease_expired', actor: 'coordinator' }).ok) out.failed++;++ if ((await store.sessions.transition(s.id, 'failed', { reason: 'lease_expired', actor: 'coordinator' })).ok) out.failed++;}// 4. generationsfor (const kind of KINDS) {-- const newest = store.workers.newestReady(kind);++ const newest = await store.workers.newestReady(kind);if (!newest) continue;-- for (const old of store.workers.olderReady(kind, newest.generation)) {-- if (store.workers.drain(old.id, now() + config.workers.drainMaxMs)) {++ for (const old of await store.workers.olderReady(kind, newest.generation)) {++ if (await store.workers.drain(old.id, now() + config.workers.drainMaxMs)) {out.drained++;log.log(`[coordinator] ${kind}#${old.generation} drains (newest ready is #${newest.generation})`);}}}// 5. outputs-- out.outputsCreated = store.outputs.ensureAutoOutputs();-- const restream = store.workers.newestReady('restream');-- if (restream) out.outputsAssigned = store.outputs.assignPending(restream);-- store.outputs.stopOrphans();++ out.outputsCreated = await store.outputs.ensureAutoOutputs();++ const restream = await store.workers.newestReady('restream');++ if (restream) out.outputsAssigned = await store.outputs.assignPending(restream);++ await store.outputs.stopOrphans();// 6. keys-- out.keysRevoked = store.definitions.expireGraceKeys();++ out.keysRevoked = await store.definitions.expireGraceKeys();// 7. recordings (requests only; the Media calls are async below)-- if (media && media.configured) out.recordingsCreated = store.recordings.ensureRequests();++ if (media && media.configured) out.recordingsCreated = await store.recordings.ensureRequests();return out;}@@@ -93,8 +93,8 @@ function createCoordinator({ rt, media, lineage = null, log = console, holder =if (ticking) return ticking;ticking = (async () => {try {-- if (!takeLease()) return { skipped: 'lease held by another coordinator' };-- const out = syncTick();++ if (!await takeLease()) return { skipped: 'lease held by another coordinator' };++ const out = await syncTick();if (media) out.recordingSteps = await store.recordings.process(media);if (lineage) out.lineage = await lineage.refresh();stats.ticks++;@@@ -124,7 +124,7 @@ function createCoordinator({ rt, media, lineage = null, log = console, holder =clearTimeout(timer);if (ticking) await ticking.catch(() => {});await rt.events.stop();-- lease.release.run(holder);++ await lease.release.run(holder);},stats: () => ({ ...stats, lineage: lineage ? lineage.stats() : null }),};@@@ -1,248 +1,42 @@'use strict';/**-- * OpenRe's durable store: one SQLite database (WAL) shared by every OpenRe process.++ * OpenRe's durable store: one PostgreSQL database (ov_openre, ADR-035) shared by every OpenRe process.** openre-api definitions, keys, destinations (owner writes)* openre-session-coordinator worker liveness, drain, output assignment, recordings, event relay* openre-rtmp-ingest sessions it admits, their leases and transitions* openre-restream-worker output state, health and logs*-- * ADR-007: SQLite stays the documented store while the platform is single-host. Several writer-- * processes is the exact trigger ADR-007 names for moving to PostgreSQL; every statement lives-- * behind server/store/* so that move changes one layer. WAL + busy_timeout keeps the writers-- * safe today (each write is one short transaction).-- *-- * Every table is created idempotently on open. Times are epoch milliseconds.++ * Several writer processes were the trigger ADR-007 named for leaving SQLite; every statement lives behind++ * server/store/*, and every transaction there is SERIALIZABLE (store/index.js), as SQLite's one writer made them.++ * The schema is migrations/NNNN_*.sql, applied at boot by the owner role. Times are epoch milliseconds.*/const fs = require('fs');const path = require('path');--const Database = require('better-sqlite3');----const SCHEMA = `--CREATE TABLE IF NOT EXISTS stream_definitions (-- id TEXT PRIMARY KEY,-- owner_subject TEXT NOT NULL,-- title TEXT NOT NULL DEFAULT 'Untitled stream',-- description TEXT NOT NULL DEFAULT '',-- protocols TEXT NOT NULL DEFAULT '["rtmp"]',-- recording_mode TEXT NOT NULL DEFAULT 'vod' CHECK (recording_mode IN ('vod', 'clips', 'none')),-- recording_visibility TEXT NOT NULL DEFAULT 'public' CHECK (recording_visibility IN ('public', 'unlisted', 'private')),-- playback_visibility TEXT NOT NULL DEFAULT 'public' CHECK (playback_visibility IN ('public', 'unlisted', 'private')),-- mirror_to_live INTEGER NOT NULL DEFAULT 0,-- state TEXT NOT NULL DEFAULT 'active' CHECK (state IN ('active', 'disabled', 'archived')),-- revision INTEGER NOT NULL DEFAULT 1,-- created_at INTEGER NOT NULL,-- updated_at INTEGER NOT NULL--);--CREATE INDEX IF NOT EXISTS idx_definitions_owner ON stream_definitions(owner_subject, state);----CREATE TABLE IF NOT EXISTS external_refs (-- definition_id TEXT NOT NULL REFERENCES stream_definitions(id) ON DELETE CASCADE,-- service TEXT NOT NULL,-- type TEXT NOT NULL,-- ref_id TEXT NOT NULL,-- label TEXT,-- created_at INTEGER NOT NULL,-- PRIMARY KEY (definition_id, service, type, ref_id)--);--CREATE INDEX IF NOT EXISTS idx_external_refs_ref ON external_refs(service, type, ref_id);----CREATE TABLE IF NOT EXISTS definition_lineage (-- definition_id TEXT PRIMARY KEY REFERENCES stream_definitions(id) ON DELETE CASCADE,-- resolution TEXT NOT NULL,-- checked_at INTEGER NOT NULL--);----CREATE TABLE IF NOT EXISTS ingest_keys (-- id TEXT PRIMARY KEY,-- definition_id TEXT NOT NULL REFERENCES stream_definitions(id) ON DELETE CASCADE,-- key_hash TEXT NOT NULL UNIQUE,-- hint TEXT NOT NULL,-- status TEXT NOT NULL CHECK (status IN ('active', 'grace', 'revoked')),-- created_by TEXT,-- created_at INTEGER NOT NULL,-- grace_until INTEGER,-- revoked_at INTEGER,-- revoked_reason TEXT,-- last_used_at INTEGER--);--CREATE INDEX IF NOT EXISTS idx_ingest_keys_definition ON ingest_keys(definition_id, status);----CREATE TABLE IF NOT EXISTS workers (-- id TEXT PRIMARY KEY,-- kind TEXT NOT NULL,-- generation INTEGER NOT NULL,-- release TEXT,-- pid INTEGER,-- host TEXT,-- state TEXT NOT NULL CHECK (state IN ('starting', 'ready', 'draining', 'stopped', 'lost')),-- endpoints TEXT NOT NULL DEFAULT '{}',-- started_at INTEGER NOT NULL,-- ready_at INTEGER,-- heartbeat_at INTEGER NOT NULL,-- drain_started_at INTEGER,-- drain_deadline INTEGER,-- stopped_at INTEGER,-- stop_reason TEXT,-- UNIQUE (kind, generation)--);--CREATE INDEX IF NOT EXISTS idx_workers_kind_state ON workers(kind, state, generation);----CREATE TABLE IF NOT EXISTS ingest_sessions (-- id TEXT PRIMARY KEY,-- definition_id TEXT NOT NULL REFERENCES stream_definitions(id),-- key_id TEXT,-- protocol TEXT NOT NULL,-- state TEXT NOT NULL CHECK (state IN ('starting', 'live', 'ending', 'ended', 'failed')),-- worker_id TEXT,-- worker_kind TEXT,-- worker_generation INTEGER,-- lease_expires_at INTEGER,-- desired_state TEXT NOT NULL DEFAULT 'run' CHECK (desired_state IN ('run', 'end')),-- end_requested_by TEXT,-- end_reason TEXT,-- failure_reason TEXT,-- media_info TEXT,-- revision INTEGER NOT NULL DEFAULT 1,-- created_at INTEGER NOT NULL,-- live_at INTEGER,-- ending_at INTEGER,-- ended_at INTEGER,-- updated_at INTEGER NOT NULL--);--CREATE INDEX IF NOT EXISTS idx_sessions_definition ON ingest_sessions(definition_id, created_at);--CREATE INDEX IF NOT EXISTS idx_sessions_state ON ingest_sessions(state);--CREATE INDEX IF NOT EXISTS idx_sessions_worker ON ingest_sessions(worker_id, state);----CREATE TABLE IF NOT EXISTS session_transitions (-- id INTEGER PRIMARY KEY AUTOINCREMENT,-- session_id TEXT NOT NULL REFERENCES ingest_sessions(id) ON DELETE CASCADE,-- from_state TEXT,-- to_state TEXT NOT NULL,-- reason TEXT,-- actor TEXT,-- at INTEGER NOT NULL--);--CREATE INDEX IF NOT EXISTS idx_transitions_session ON session_transitions(session_id, id);++const { createDb } = require('openvibe-sdk/db');--CREATE TABLE IF NOT EXISTS destinations (-- id TEXT PRIMARY KEY,-- definition_id TEXT NOT NULL REFERENCES stream_definitions(id) ON DELETE CASCADE,-- platform TEXT NOT NULL CHECK (platform IN ('youtube', 'twitch', 'kick', 'custom')),-- name TEXT,-- server_url TEXT NOT NULL,-- stream_key_enc TEXT,-- key_hint TEXT,-- srt_passphrase_enc TEXT,-- srt_latency_ms INTEGER,-- enabled INTEGER NOT NULL DEFAULT 1,-- auto_start INTEGER NOT NULL DEFAULT 1,-- quality_preset TEXT NOT NULL DEFAULT 'auto',-- custom_video_bitrate INTEGER,-- custom_audio_bitrate INTEGER,-- custom_fps INTEGER,-- custom_encoder_preset TEXT,-- hold_reason TEXT,-- consecutive_failures INTEGER NOT NULL DEFAULT 0,-- cooldown_until INTEGER,-- last_error TEXT,-- last_failed_at INTEGER,-- created_at INTEGER NOT NULL,-- updated_at INTEGER NOT NULL--);--CREATE INDEX IF NOT EXISTS idx_destinations_definition ON destinations(definition_id);++const MIGRATIONS = path.join(__dirname, '..', 'migrations');++const DEV_PGLITE = path.join(__dirname, '..', 'data', 'pglite');--CREATE TABLE IF NOT EXISTS outputs (-- id TEXT PRIMARY KEY,-- session_id TEXT NOT NULL REFERENCES ingest_sessions(id),-- destination_id TEXT NOT NULL REFERENCES destinations(id) ON DELETE CASCADE,-- desired TEXT NOT NULL DEFAULT 'run' CHECK (desired IN ('run', 'stop')),-- state TEXT NOT NULL DEFAULT 'pending' CHECK (state IN ('pending', 'starting', 'live', 'error', 'failed', 'stopped')),-- worker_id TEXT,-- worker_generation INTEGER,-- restart_attempts INTEGER NOT NULL DEFAULT 0,-- next_restart_at INTEGER,-- ever_live INTEGER NOT NULL DEFAULT 0,-- last_error TEXT,-- progress TEXT,-- started_at INTEGER,-- live_at INTEGER,-- ended_at INTEGER,-- created_at INTEGER NOT NULL,-- updated_at INTEGER NOT NULL,-- UNIQUE (session_id, destination_id)--);--CREATE INDEX IF NOT EXISTS idx_outputs_worker ON outputs(worker_id, state);--CREATE INDEX IF NOT EXISTS idx_outputs_session ON outputs(session_id);----CREATE TABLE IF NOT EXISTS output_logs (-- id INTEGER PRIMARY KEY AUTOINCREMENT,-- output_id TEXT,-- destination_id TEXT NOT NULL,-- level TEXT NOT NULL CHECK (level IN ('info', 'warn', 'error')),-- message TEXT NOT NULL,-- at INTEGER NOT NULL--);--CREATE INDEX IF NOT EXISTS idx_output_logs_output ON output_logs(output_id, id);--CREATE INDEX IF NOT EXISTS idx_output_logs_destination ON output_logs(destination_id, id);----CREATE TABLE IF NOT EXISTS recordings (-- id TEXT PRIMARY KEY,-- session_id TEXT NOT NULL UNIQUE REFERENCES ingest_sessions(id),-- mode TEXT NOT NULL CHECK (mode IN ('vod', 'clips')),-- state TEXT NOT NULL CHECK (state IN ('pending', 'requested', 'recording', 'finalizing', 'finalized', 'failed', 'cancelled')),-- media_app TEXT,-- media_vod_id TEXT,-- attempts INTEGER NOT NULL DEFAULT 0,-- next_attempt_at INTEGER NOT NULL DEFAULT 0,-- last_error TEXT,-- created_at INTEGER NOT NULL,-- updated_at INTEGER NOT NULL--);--CREATE INDEX IF NOT EXISTS idx_recordings_state ON recordings(state, next_attempt_at);----CREATE TABLE IF NOT EXISTS migration_map (-- source_system TEXT NOT NULL,-- source_type TEXT NOT NULL,-- source_id TEXT NOT NULL,-- target_type TEXT,-- target_id TEXT,-- status TEXT NOT NULL CHECK (status IN ('imported', 'held', 'excluded')),-- reason TEXT,-- imported_at INTEGER NOT NULL,-- PRIMARY KEY (source_system, source_type, source_id)--);----CREATE TABLE IF NOT EXISTS leases (-- name TEXT PRIMARY KEY,-- holder TEXT NOT NULL,-- expires_at INTEGER NOT NULL--);--`;----function openDb(dbPath) {-- if (dbPath !== ':memory:') fs.mkdirSync(path.dirname(path.resolve(dbPath)), { recursive: true });-- const db = new Database(dbPath);-- // busy_timeout first: several OpenRe processes open the database at the same time on boot,-- // and every statement after this one (the WAL switch, the schema) may have to wait for a lock.-- db.pragma('busy_timeout = 5000');-- db.pragma('journal_mode = WAL');-- db.pragma('foreign_keys = ON');-- db.pragma('synchronous = NORMAL');-- db.exec(SCHEMA);-- return db;++/**++ * The serving handle: DATABASE_URL through PgBouncer, after the migrations ran as the owner (DATABASE_DIRECT_URL).++ * Without DATABASE_URL (development, tests), an embedded PGlite database: in OPENRE_PGLITE_DIR or data/pglite, or in++ * memory with { memory: true }. PGlite is one process: run the other OpenRe processes against PostgreSQL.++ */++async function openDb(config, { memory = false, log = console, registry } = {}) {++ const quiet = { ...log, info() {}, log() {} };++ if (!config.db.url) {++ if (config.isProduction) throw new Error('DATABASE_URL is not set: production serves from PostgreSQL (OpenVibe.Host roles/data add-service.sh openre)');++ const dir = memory ? null : (config.db.pgliteDir || DEV_PGLITE);++ if (dir) fs.mkdirSync(dir, { recursive: true });++ const db = createDb({ pglite: dir || true, service: 'openre', registry, log });++ await db.migrate({ dir: MIGRATIONS, log: quiet });++ return db;++ }++ if (!config.db.directUrl) throw new Error('DATABASE_DIRECT_URL is not set: migrations run with the owner role on a direct connection');++ const owner = createDb({ url: config.db.directUrl, service: 'openre-migrate', max: 1, log });++ try { await owner.migrate({ dir: MIGRATIONS, log: quiet }); } finally { await owner.close(); }++ return createDb({ url: config.db.url, service: 'openre', registry, log });}--module.exports = { openDb, SCHEMA };++module.exports = { openDb, MIGRATIONS };@@@ -2,7 +2,7 @@/*** OpenRe → OpenVibe.Events through the openvibe-sdk transactional outbox (ADR-004).*-- * Every OpenRe process can enqueue (the row commits in the same SQLite transaction as the change++ * Every OpenRe process can enqueue (the row commits in the same transaction as the change* it describes); only the session coordinator runs the relay that publishes rows with OpenRe's* service token (audience openvibe.events, capability events.event.publish). Events down, Network* down or no credentials yet: rows wait and are retried; nothing in the transport path waits.@@@ -19,7 +19,7 @@*/const { createClient } = require('openvibe-sdk/core');const { createServiceTokenClient } = require('openvibe-sdk/auth');--const { createEventsClient, createOutbox } = require('openvibe-sdk/events');++const { createEventsClient, createPgOutbox } = require('openvibe-sdk/events');const TYPES = Object.freeze({sessionStarted: 'openre.session.started',@@@ -58,7 +58,7 @@ function createEvents({ config, db, fetchImpl = globalThis.fetch, now = () => Da});const eventsClient = createEventsClient(client, { source: 'openre' });let lastError = null;-- const outbox = createOutbox(db, {++ const outbox = createPgOutbox(db, {events: eventsClient,intervalMs: config.events.intervalMs,now,@@@ -68,15 +68,15 @@ function createEvents({ config, db, fetchImpl = globalThis.fetch, now = () => DalastError = msg;},});-- outbox.ensureSchema();return {outbox,configured,-- enqueue: (envelope) => outbox.enqueue(envelope),++ // Joins the caller's transaction (the SDK handle is ambient): the row commits with the change it describes.++ enqueue: async (envelope) => await outbox.enqueue(db, envelope),start() { if (configured) outbox.start(); return configured; },stop: () => outbox.stop(),-- status: () => ({ configured, pending: outbox.pending(), rejected: outbox.rejected(), last_error: lastError }),++ status: async () => ({ configured, pending: await outbox.pending(), rejected: await outbox.rejected(), last_error: lastError }),};}@@@ -6,7 +6,7 @@* end requests) and serves the UI. Restarting or redeploying it never touches a worker process,* a session or an ffmpeg output; see test/api-restart.test.js.*-- * start() is what the tests use too: it takes a config plus injectable clock/fetch/log.++ * start() is what the tests use too: it takes a config plus injectable clock/fetch/log (and a database handle).*/const path = require('path');const { load } = require('./config');@@@ -14,9 +14,9 @@ const { openRuntime } = require('./store');const { createKeyStore, createAuth } = require('./auth');const { createApp } = require('./app');--async function start({ config, clock, fetchImpl = globalThis.fetch, log = console, listen = true } = {}) {++async function start({ config, clock, fetchImpl = globalThis.fetch, log = console, listen = true, db } = {}) {config = config || load();-- const rt = openRuntime({ config, clock, fetchImpl, log });++ const rt = await openRuntime({ config, clock, fetchImpl, log, db });const keys = createKeyStore({ urls: [config.networkInternalUrl, config.networkUrl], pem: config.networkPublicKey, fetchImpl, log });const auth = createAuth({ config, keys });const app = createApp({ rt, auth, keys, log, fetchImpl });@@@ -37,7 +37,8 @@ async function start({ config, clock, fetchImpl = globalThis.fetch, log = consolserver.closeAllConnections?.();await new Promise(resolve => server.close(() => resolve()));}-- rt.db.close();++ if (app.locals.valkey) await app.locals.valkey.close().catch(() => {});++ await rt.db.close();}return { config, rt, keys, keyLoaded, auth, app, server, close };@@@ -43,10 +43,10 @@ function createLineage({ db, config, env = process.env, fetchImpl = globalThis.fasync function refresh() {if (!enabled) return { skipped: 'off' };const out = { checked: 0, resolved: 0 };-- for (const def of q.due.all(now() - maxAgeMs, batch)) {++ for (const def of await q.due.all(now() - maxAgeMs, batch)) {try {const r = await resolveOne(def);-- q.save.run(def.id, JSON.stringify(r), now());++ await q.save.run(def.id, JSON.stringify(r), now());out.checked++; stats.refreshed++;if (r.status === 'resolved') { out.resolved++; stats.resolved++; } else stats.unresolved++;} catch (err) {@@@ -56,10 +56,10 @@ function createMediaClient({ config, fetchImpl = globalThis.fetch }) {return {configured,appId: config.media.appId,-- createVod: (fields) => request('POST', '/vods', { body: fields }),-- ingestRtmp: (vodId, rtmpUrl) => request('POST', `/vods/${encodeURIComponent(vodId)}/ingest/rtmp`, { body: { rtmp_url: rtmpUrl } }),-- finalizeVod: (vodId) => request('POST', `/vods/${encodeURIComponent(vodId)}/finalize`, { timeoutMs: 30000 }),-- deleteVod: (vodId) => request('DELETE', `/vods/${encodeURIComponent(vodId)}`),++ createVod: async (fields) => await request('POST', '/vods', { body: fields }),++ ingestRtmp: async (vodId, rtmpUrl) => await request('POST', `/vods/${encodeURIComponent(vodId)}/ingest/rtmp`, { body: { rtmp_url: rtmpUrl } }),++ finalizeVod: async (vodId) => await request('POST', `/vods/${encodeURIComponent(vodId)}/finalize`, { timeoutMs: 30000 }),++ deleteVod: async (vodId) => await request('DELETE', `/vods/${encodeURIComponent(vodId)}`),};}@@@ -18,8 +18,8 @@const { validateDestinationUrl } = require('./destination-url');--function columns(db, table) {-- try { return new Set(db.prepare(`PRAGMA table_info(${table})`).all().map(c => c.name)); } catch { return new Set(); }++async function columns(db, table) {++ try { return new Set((await db.prepare(`PRAGMA table_info(${table})`).all()).map(c => c.name)); } catch { return new Set(); }}function recordingModeOf(ms, channel) {@@@ -38,21 +38,21 @@ function visibilityOf(ms, channel) {* migrate({ liveDb, rt, apply, onlySlots }) -> report* report = { slots: [...], destinations: [...], channels: Map username → {...}, counts }*/--function migrate({ liveDb, rt, apply = false, onlySlots = null, now = () => Date.now() }) {++async function migrate({ liveDb, rt, apply = false, onlySlots = null, now = () => Date.now() }) {const { db, store, config } = rt;-- const msCols = columns(liveDb, 'managed_streams');++ const msCols = await columns(liveDb, 'managed_streams');if (!msCols.size) throw new Error('the Live snapshot has no managed_streams table');-- const rdCols = columns(liveDb, 'restream_destinations');-- const laCols = columns(liveDb, 'linked_accounts');-- const chCols = columns(liveDb, 'channels');++ const rdCols = await columns(liveDb, 'restream_destinations');++ const laCols = await columns(liveDb, 'linked_accounts');++ const chCols = await columns(liveDb, 'channels');const mapGet = db.prepare("SELECT * FROM migration_map WHERE source_system = 'live' AND source_type = ? AND source_id = ?");const mapPut = db.prepare(`INSERT INTO migration_map (source_system, source_type, source_id, target_type, target_id, status, reason, imported_at)VALUES ('live', @source_type, @source_id, @target_type, @target_id, @status, @reason, @at)ON CONFLICT(source_system, source_type, source_id) DO UPDATE SET target_type = @target_type, target_id = @target_id, status = @status, reason = @reason, imported_at = @at`);-- const record = (row) => { if (apply) mapPut.run({ target_type: null, target_id: null, reason: null, ...row, at: now() }); };++ const record = async (row) => { if (apply) await mapPut.run({ target_type: null, target_id: null, reason: null, ...row, at: now() }); };-- const slots = liveDb.prepare(`SELECT ms.*, u.username, u.display_name, u.is_banned FROM managed_streams ms JOIN users u ON u.id = ms.user_id ORDER BY ms.user_id, ms.id`).all()++ const slots = (await liveDb.prepare(`SELECT ms.*, u.username, u.display_name, u.is_banned FROM managed_streams ms JOIN users u ON u.id = ms.user_id ORDER BY ms.user_id, ms.id`).all()).filter(ms => !onlySlots || onlySlots.includes(Number(ms.id)));const subjectOf = laCols.has('subject_id')? liveDb.prepare("SELECT subject_id FROM linked_accounts WHERE service = 'network' AND user_id = ? AND subject_id IS NOT NULL")@@@ -73,11 +73,11 @@ function migrate({ liveDb, rt, apply = false, onlySlots = null, now = () => Datefor (const ms of slots) {const ch = channel(ms);-- const prior = mapGet.get('managed_stream', String(ms.id));++ const prior = await mapGet.get('managed_stream', String(ms.id));const entry = { live_id: ms.id, slug: ms.slug || null, title: ms.title, protocol: ms.protocol, streaming_method: ms.streaming_method || null, definition_id: null, status: null, reason: null, destinations: [] };ch.slots.push(entry);-- if (prior && prior.status === 'imported' && store.definitions.get(prior.target_id)) {++ if (prior && prior.status === 'imported' && await store.definitions.get(prior.target_id)) {entry.definition_id = prior.target_id;entry.status = 'imported';entry.reason = 'already imported';@@@ -87,12 +87,12 @@ function migrate({ liveDb, rt, apply = false, onlySlots = null, now = () => Dateif (!subject) {entry.status = 'held';entry.reason = `Live user ${ms.user_id} has no canonical subject yet (it is recorded when they next sign in to Live); the slot is imported on a later run`;-- record({ source_type: 'managed_stream', source_id: String(ms.id), status: 'held', reason: entry.reason });++ await record({ source_type: 'managed_stream', source_id: String(ms.id), status: 'held', reason: entry.reason });count('held');} else if (ms.is_banned) {entry.status = 'excluded';entry.reason = 'the Live account is banned';-- record({ source_type: 'managed_stream', source_id: String(ms.id), status: 'excluded', reason: entry.reason });++ await record({ source_type: 'managed_stream', source_id: String(ms.id), status: 'excluded', reason: entry.reason });count('excluded');} else {const chRow = channelOf ? channelOf.get(ms.user_id) : null;@@@ -113,10 +113,10 @@ function migrate({ liveDb, rt, apply = false, onlySlots = null, now = () => Date};entry.recording_mode = fields.recording_mode;if (apply) {-- const existing = store.definitions.findByRef('live', 'managed_stream', String(ms.id));-- const def = existing || store.definitions.create(fields).definition; // the new key is discarded unseen++ const existing = await store.definitions.findByRef('live', 'managed_stream', String(ms.id));++ const def = existing || (await store.definitions.create(fields)).definition; // the new key is discarded unseenentry.definition_id = def.id;-- record({ source_type: 'managed_stream', source_id: String(ms.id), target_type: 'stream_definition', target_id: def.id, status: 'imported', reason: 'new key issued, old key not imported' });++ await record({ source_type: 'managed_stream', source_id: String(ms.id), target_type: 'stream_definition', target_id: def.id, status: 'imported', reason: 'new key issued, old key not imported' });}entry.status = 'imported';entry.reason = 'new key issued, old key not imported';@@@ -127,28 +127,28 @@ function migrate({ liveDb, rt, apply = false, onlySlots = null, now = () => Date// Destinations: the slot's own, plus unbound ones when the user has exactly one slot.const own = destsOfSlot ? destsOfSlot.all(ms.id) : [];const unbound = unboundOfUser ? unboundOfUser.all(ms.user_id) : [];-- const oneSlot = slotsOfUser.get(ms.user_id).n === 1;++ const oneSlot = (await slotsOfUser.get(ms.user_id)).n === 1;const candidates = [...own.map(d => ({ d, unbound: false })), ...unbound.map(d => ({ d, unbound: true }))];for (const { d, unbound: isUnbound } of candidates) {const di = { live_id: d.id, platform: d.platform, name: d.name || d.platform, status: null, reason: null, destination_id: null, oauth_linked: Boolean(d.connection_id) };entry.destinations.push(di);report.destinations.push(di);-- const priorD = mapGet.get('restream_destination', String(d.id));-- if (priorD && priorD.target_id && store.outputs.destinationRow(priorD.target_id)) {++ const priorD = await mapGet.get('restream_destination', String(d.id));++ if (priorD && priorD.target_id && await store.outputs.destinationRow(priorD.target_id)) {di.status = priorD.status; di.reason = priorD.reason || 'already imported'; di.destination_id = priorD.target_id; count('skipped');continue;}if (isUnbound && !oneSlot) {di.status = 'held';di.reason = 'not bound to a slot and the user has several slots (Live pushed unbound destinations from every slot); bind it in Live or add it on openre.stream';-- record({ source_type: 'restream_destination', source_id: String(d.id), status: 'held', reason: di.reason });++ await record({ source_type: 'restream_destination', source_id: String(d.id), status: 'held', reason: di.reason });count('held');continue;}if (entry.status !== 'imported') {di.status = 'held';di.reason = `its slot is ${entry.status}`;-- record({ source_type: 'restream_destination', source_id: String(d.id), status: 'held', reason: di.reason });++ await record({ source_type: 'restream_destination', source_id: String(d.id), status: 'held', reason: di.reason });count('held');continue;}@@@ -174,12 +174,12 @@ function migrate({ liveDb, rt, apply = false, onlySlots = null, now = () => Dateif (hold) {// Keep the (sealed) key so the owner only has to fix the URL; the row stays// disabled until they do. The original URL is kept in the hold reason.-- created = store.outputs.createDestination(entry.definition_id, { ...input, server_url: 'rtmp://held.invalid/app', enabled: false }, { hold_reason: `${hold} (was ${String(d.server_url || '').replace(/\/[^/]*$/, '/…')})` });++ created = await store.outputs.createDestination(entry.definition_id, { ...input, server_url: 'rtmp://held.invalid/app', enabled: false }, { hold_reason: `${hold} (was ${String(d.server_url || '').replace(/\/[^/]*$/, '/…')})` });} else {-- created = store.outputs.createDestination(entry.definition_id, input);++ created = await store.outputs.createDestination(entry.definition_id, input);}di.destination_id = created.id;-- record({ source_type: 'restream_destination', source_id: String(d.id), target_type: 'destination', target_id: created.id, status: hold ? 'held' : 'imported', reason: hold || null });++ await record({ source_type: 'restream_destination', source_id: String(d.id), target_type: 'destination', target_id: created.id, status: hold ? 'held' : 'imported', reason: hold || null });}di.status = hold ? 'held' : 'imported';di.reason = hold;@@@ -71,34 +71,34 @@ function createDefinitions({ db, config, events, clock }) {expiredGrace: db.prepare("SELECT id FROM ingest_keys WHERE status = 'grace' AND grace_until <= ?"),};-- function row(id) {-- const d = q.get.get(id);++ async function row(id) {++ const d = await q.get.get(id);if (!d) return null;return {...d,protocols: parseJson(d.protocols, ['rtmp']),mirror_to_live: Boolean(d.mirror_to_live),-- external_refs: q.refs.all(id),++ external_refs: await q.refs.all(id),};}-- function get(id) {-- return isId('stream', id) ? row(id) : null;++ async function get(id) {++ return isId('stream', id) ? await row(id) : null;}-- function list({ owner_subject, state, limit = 100 } = {}) {++ async function list({ owner_subject, state, limit = 100 } = {}) {const where = [];const args = [];if (owner_subject) { where.push('owner_subject = ?'); args.push(owner_subject); }if (state) { where.push('state = ?'); args.push(state); } else where.push("state != 'archived'");-- const rows = db.prepare(`SELECT id FROM stream_definitions ${where.length ? `WHERE ${where.join(' AND ')}` : ''}++ const rows = await db.prepare(`SELECT id FROM stream_definitions ${where.length ? `WHERE ${where.join(' AND ')}` : ''}ORDER BY created_at DESC LIMIT ?`).all(...args, Math.min(Math.max(limit, 1), 500));-- return rows.map(r => row(r.id));++ return (await Promise.all(rows.map(async r => await row(r.id))));}-- function findByRef(service, type, refId) {-- const r = q.byRef.get(String(service), String(type), String(refId));-- return r ? row(r.definition_id) : null;++ async function findByRef(service, type, refId) {++ const r = await q.byRef.get(String(service), String(type), String(refId));++ return r ? await row(r.definition_id) : null;}function normalize(input, base = {}) {@@@ -125,10 +125,10 @@ function createDefinitions({ db, config, events, clock }) {}/** Issue a new active key (inside the caller's transaction). Returns the plain key once. */-- function issueKey(definitionId, createdBy) {++ async function issueKey(definitionId, createdBy) {const plain = newIngestKey();const id = newId('key', now());-- q.insertKey.run(id, definitionId, hashIngestKey(plain), hintOf(plain), createdBy || null, now());++ await q.insertKey.run(id, definitionId, hashIngestKey(plain), hintOf(plain), createdBy || null, now());return { id, key: plain, hint: hintOf(plain) };}@@@ -136,47 +136,47 @@ function createDefinitions({ db, config, events, clock }) {* create({ owner_subject, title, ..., external_refs, created_by }) -> { definition, key: { id, key, hint } }* The first ingest key is issued with the definition; its plain value is only in this return.*/-- function create(input) {++ async function create(input) {if (!ids.isSubjectId('user', input.owner_subject)) throw new StoreError(400, 'openre.invalid_owner', 'owner_subject must be a usr_… subject id');const given = Object.fromEntries(Object.entries(input).filter(([, v]) => v !== undefined && v !== null && v !== ''));const fields = normalize({ protocols: ['rtmp'], recording_mode: 'vod', recording_visibility: 'public', playback_visibility: 'public', mirror_to_live: false, ...given });const refs = validateRefs(input.external_refs);const id = newId('stream', now());-- return db.transaction(() => {++ return await db.tx(async () => {for (const r of refs) {if (SHARED_REF_TYPES.has(r.type)) continue;-- const taken = q.byRef.get(r.service, r.type, r.id);-- if (taken && row(taken.definition_id).state !== 'archived') throw new StoreError(409, 'openre.ref_taken', `${r.service}:${r.type}:${r.id} already belongs to another stream definition`);++ const taken = await q.byRef.get(r.service, r.type, r.id);++ if (taken && (await row(taken.definition_id)).state !== 'archived') throw new StoreError(409, 'openre.ref_taken', `${r.service}:${r.type}:${r.id} already belongs to another stream definition`);}-- q.insert.run({ id, owner_subject: input.owner_subject, description: '', ...fields, now: now() });-- for (const r of refs) q.insertRef.run(id, r.service, r.type, r.id, r.label, now());-- const key = issueKey(id, input.created_by);-- return { definition: row(id), key };-- })();++ await q.insert.run({ id, owner_subject: input.owner_subject, description: '', ...fields, now: now() });++ for (const r of refs) await q.insertRef.run(id, r.service, r.type, r.id, r.label, now());++ const key = await issueKey(id, input.created_by);++ return { definition: await row(id), key };++ });}-- function update(id, input) {-- const current = get(id);++ async function update(id, input) {++ const current = await get(id);if (!current || current.state === 'archived') throw new StoreError(404, 'openre.stream_not_found', 'no such stream definition');const fields = normalize(input, current);const keys = Object.keys(fields);if (!keys.length) return current;-- db.prepare(`UPDATE stream_definitions SET ${keys.map(k => `${k} = @${k}`).join(', ')}, revision = revision + 1, updated_at = @now WHERE id = @id`)++ await db.prepare(`UPDATE stream_definitions SET ${keys.map(k => `${k} = @${k}`).join(', ')}, revision = revision + 1, updated_at = @now WHERE id = @id`).run({ ...fields, id, now: now() });-- return row(id);++ return await row(id);}-- function archive(id) {-- const current = get(id);++ async function archive(id) {++ const current = await get(id);if (!current || current.state === 'archived') throw new StoreError(404, 'openre.stream_not_found', 'no such stream definition');-- const live = db.prepare("SELECT id FROM ingest_sessions WHERE definition_id = ? AND state IN ('starting', 'live', 'ending')").get(id);++ const live = await db.prepare("SELECT id FROM ingest_sessions WHERE definition_id = ? AND state IN ('starting', 'live', 'ending')").get(id);if (live) throw new StoreError(409, 'openre.stream_live', 'end the live session before archiving this stream');-- db.transaction(() => {-- db.prepare("UPDATE stream_definitions SET state = 'archived', revision = revision + 1, updated_at = ? WHERE id = ?").run(now(), id);-- for (const k of q.activeKeys.all(id)) q.revokeKey.run(now(), 'archived', k.id);-- db.prepare("UPDATE ingest_keys SET status = 'revoked', revoked_at = ?, revoked_reason = 'archived' WHERE definition_id = ? AND status = 'grace'").run(now(), id);-- })();-- return row(id);++ await db.tx(async () => {++ await db.prepare("UPDATE stream_definitions SET state = 'archived', revision = revision + 1, updated_at = ? WHERE id = ?").run(now(), id);++ for (const k of await q.activeKeys.all(id)) await q.revokeKey.run(now(), 'archived', k.id);++ await db.prepare("UPDATE ingest_keys SET status = 'revoked', revoked_at = ?, revoked_reason = 'archived' WHERE definition_id = ? AND status = 'grace'").run(now(), id);++ });++ return await row(id);}/**@@@ -185,19 +185,19 @@ function createDefinitions({ db, config, events, clock }) {* coordinator when the grace ends). Sessions already running are never cut by a rotation* unless end_sessions is set (the caller then asks the owning worker to end them).*/-- function rotateKey(id, { grace_seconds = 0, rotated_by = null, reason = 'rotated' } = {}) {-- const current = get(id);++ async function rotateKey(id, { grace_seconds = 0, rotated_by = null, reason = 'rotated' } = {}) {++ const current = await get(id);if (!current || current.state === 'archived') throw new StoreError(404, 'openre.stream_not_found', 'no such stream definition');const grace = Math.max(0, Math.min(Number(grace_seconds) || 0, 7 * 24 * 3600));-- return db.transaction(() => {-- const old = q.activeKeys.all(id);++ return await db.tx(async () => {++ const old = await q.activeKeys.all(id);const graceUntil = grace ? now() + grace * 1000 : null;for (const k of old) {-- if (graceUntil) q.graceKey.run(graceUntil, k.id);-- else q.revokeKey.run(now(), reason, k.id);++ if (graceUntil) await q.graceKey.run(graceUntil, k.id);++ else await q.revokeKey.run(now(), reason, k.id);}-- const key = issueKey(id, rotated_by);-- events.enqueue({++ const key = await issueKey(id, rotated_by);++ await events.enqueue({event_type: TYPES.keyRotated,actor: rotated_by && /^usr_/.test(rotated_by) ? { type: 'user', id: rotated_by } : { type: 'service', id: rotated_by ? String(rotated_by).replace(/^svc:/, '') : 'openre' },subject: { type: 'stream', id, revision: current.revision },@@@ -214,18 +214,18 @@ function createDefinitions({ db, config, events, clock }) {},});return { key, retired: old.map(k => ({ id: k.id, hint: k.hint, status: graceUntil ? 'grace' : 'revoked' })), grace_until: graceUntil };-- })();++ });}/** Revoke keys whose grace period is over (coordinator). */-- function expireGraceKeys() {-- const rows = q.expiredGrace.all(now());-- for (const r of rows) q.revokeKey.run(now(), 'grace_ended', r.id);++ async function expireGraceKeys() {++ const rows = await q.expiredGrace.all(now());++ for (const r of rows) await q.revokeKey.run(now(), 'grace_ended', r.id);return rows.length;}-- function keys(id) {-- return q.keysOf.all(id);++ async function keys(id) {++ return await q.keysOf.all(id);}/**@@@ -233,24 +233,24 @@ function createDefinitions({ db, config, events, clock }) {* answered inside node-media-server's prePublish handler. Only OpenRe's own hashed keys count:* this never looks at any other service's database.*/-- function resolveIngestKey(plain, protocol) {++ async function resolveIngestKey(plain, protocol) {if (!isIngestKeyShape(plain)) return { error: 'malformed_key' };-- const key = q.keyByHash.get(hashIngestKey(plain));++ const key = await q.keyByHash.get(hashIngestKey(plain));if (!key) return { error: 'unknown_key' };if (key.status === 'revoked') return { error: 'revoked_key' };if (key.status === 'grace' && !(key.grace_until > now())) return { error: 'revoked_key' };-- const definition = row(key.definition_id);++ const definition = await row(key.definition_id);if (!definition || definition.state !== 'active') return { error: 'stream_disabled' };if (!definition.protocols.includes(protocol)) return { error: 'protocol_not_allowed' };-- q.touchKey.run(now(), key.id);++ await q.touchKey.run(now(), key.id);return { definition, key: { id: key.id, hint: key.hint, status: key.status } };}/** What an encoder needs: server URL and which key (by hint) is current. Never the key. */-- function ingestEndpoints(definition) {++ async function ingestEndpoints(definition) {const r = config.rtmp;const portSuffix = r.publicPort === 1935 ? '' : `:${r.publicPort}`;-- const active = keys(definition.id).find(k => k.status === 'active');++ const active = (await keys(definition.id)).find(k => k.status === 'active');const out = {};if (definition.protocols.includes('rtmp')) {out.rtmp = {@@@ -2,7 +2,7 @@/*** One entry point for every process: open the database, the event outbox and the store modules.*-- * const rt = openRuntime({ config }); // rt.db, rt.events, rt.store.{definitions,sessions,...}++ * const rt = await openRuntime({ config }); // rt.db, rt.events, rt.store.{definitions,sessions,...}*/const { openDb } = require('../db');const { createEvents } = require('../events');@@@ -13,7 +13,25 @@ const { createSessions } = require('./sessions');const { createOutputs } = require('./outputs');const { createRecordings } = require('./recordings');++/**++ * The handle every store gets: its transactions are SERIALIZABLE. Several OpenRe processes change the same rows (a++ * publish admitted by ingest while the coordinator fails a lost worker's session), and SQLite's one writer used to++ * order them; PostgreSQL detects the conflict and the SDK runs the loser again (40001 is retried).++ */++function serializable(db) {++ if (db.serializable) return db;++ return new Proxy(db, {++ get(target, key) {++ if (key === 'serializable') return true;++ if (key === 'tx') return async (fn, opts = {}) => await target.tx(fn, { isolation: 'serializable', ...opts });++ const v = target[key];++ return typeof v === 'function' ? v.bind(target) : v;++ },++ });++}++function createStore({ db, config, events, clock, box, log = console }) {++ db = serializable(db);const definitions = createDefinitions({ db, config, events, clock });const workers = createWorkers({ db, config, clock });const sessions = createSessions({ db, config, events, clock, definitions, workers });@@@ -22,12 +40,12 @@ function createStore({ db, config, events, clock, box, log = console }) {return { definitions, workers, sessions, outputs, recordings };}--function openRuntime({ config, clock = { now: () => Date.now() }, fetchImpl = globalThis.fetch, log = console, db } = {}) {-- db = db || openDb(config.dbPath);++async function openRuntime({ config, clock = { now: () => Date.now() }, fetchImpl = globalThis.fetch, log = console, db } = {}) {++ db = serializable(db || await openDb(config, { log }));const events = createEvents({ config, db, fetchImpl, now: () => clock.now(), log });const box = createBox({ key: config.secretsKey, previous: config.secretsKeyPrevious });const store = createStore({ db, config, events, clock, box, log });return { db, events, box, store, clock, config };}--module.exports = { openRuntime, createStore };++module.exports = { openRuntime, createStore, serializable };@@@ -40,7 +40,7 @@ function createOutputs({ db, config, events, clock, box, definitions, sessions }WHERE state IN ${OPEN} AND (desired = 'stop' AND (worker_id IS NULL OR worker_id IN (SELECT id FROM workers WHERE state IN ('stopped', 'lost'))))`),insertLog: db.prepare('INSERT INTO output_logs (output_id, destination_id, level, message, at) VALUES (?, ?, ?, ?, ?)'),pruneLogs: db.prepare(`DELETE FROM output_logs WHERE id IN (SELECT id FROM output_logs WHERE destination_id = ? AND-- (output_id IS ? OR output_id = ?) ORDER BY id DESC LIMIT -1 OFFSET ?)`),++ (output_id IS NOT DISTINCT FROM ? OR output_id = ?) ORDER BY id DESC LIMIT ALL OFFSET ?)`),logsOfOutput: db.prepare('SELECT level, message, at FROM output_logs WHERE output_id = ? ORDER BY id DESC LIMIT ?'),logsOfDest: db.prepare('SELECT output_id, level, message, at FROM output_logs WHERE destination_id = ? ORDER BY id DESC LIMIT ?'),};@@@ -126,21 +126,21 @@ function createOutputs({ db, config, events, clock, box, definitions, sessions }return f;}-- function createDestination(definitionId, input, { hold_reason = null } = {}) {-- const def = definitions.get(definitionId);++ async function createDestination(definitionId, input, { hold_reason = null } = {}) {++ const def = await definitions.get(definitionId);if (!def || def.state === 'archived') throw new StoreError(404, 'openre.stream_not_found', 'no such stream definition');-- if (q.countOf.get(definitionId).n >= config.outputs.maxPerStream) throw new StoreError(409, 'openre.too_many_destinations', `at most ${config.outputs.maxPerStream} destinations per stream`);++ if ((await q.countOf.get(definitionId)).n >= config.outputs.maxPerStream) throw new StoreError(409, 'openre.too_many_destinations', `at most ${config.outputs.maxPerStream} destinations per stream`);const f = destFields(input, { creating: true });const id = newId('destination', now());const row = { id, definition_id: definitionId, enabled: 1, auto_start: 1, quality_preset: 'auto', ...f, hold_reason, created_at: now(), updated_at: now() };if (hold_reason) row.enabled = 0;const cols = Object.keys(row);-- db.prepare(`INSERT INTO destinations (${cols.join(', ')}) VALUES (${cols.map(c => `@${c}`).join(', ')})`).run(row);-- return publicDest(q.getDest.get(id));++ await db.prepare(`INSERT INTO destinations (${cols.join(', ')}) VALUES (${cols.map(c => `@${c}`).join(', ')})`).run(row);++ return publicDest(await q.getDest.get(id));}-- function updateDestination(id, input) {-- const d = q.getDest.get(id);++ async function updateDestination(id, input) {++ const d = await q.getDest.get(id);if (!d) throw new StoreError(404, 'openre.destination_not_found', 'no such destination');const f = destFields(input, { creating: false });// Re-enabling a held destination is an explicit owner decision: the hold is lifted only@@@ -153,42 +153,42 @@ function createOutputs({ db, config, events, clock, box, definitions, sessions }if (!Object.keys(f).length) return publicDest(d);f.updated_at = now();const cols = Object.keys(f);-- db.prepare(`UPDATE destinations SET ${cols.map(c => `${c} = @${c}`).join(', ')} WHERE id = @id`).run({ ...f, id });-- return publicDest(q.getDest.get(id));++ await db.prepare(`UPDATE destinations SET ${cols.map(c => `${c} = @${c}`).join(', ')} WHERE id = @id`).run({ ...f, id });++ return publicDest(await q.getDest.get(id));}-- function deleteDestination(id) {-- const d = q.getDest.get(id);++ async function deleteDestination(id) {++ const d = await q.getDest.get(id);if (!d) throw new StoreError(404, 'openre.destination_not_found', 'no such destination');-- db.transaction(() => {-- db.prepare(`UPDATE outputs SET desired = 'stop', updated_at = ? WHERE destination_id = ? AND state IN ${OPEN}`).run(now(), id);-- const running = db.prepare(`SELECT 1 FROM outputs WHERE destination_id = ? AND state IN ${OPEN} AND worker_id IS NOT NULL`).get(id);++ await db.tx(async () => {++ await db.prepare(`UPDATE outputs SET desired = 'stop', updated_at = ? WHERE destination_id = ? AND state IN ${OPEN}`).run(now(), id);++ const running = await db.prepare(`SELECT 1 FROM outputs WHERE destination_id = ? AND state IN ${OPEN} AND worker_id IS NOT NULL`).get(id);if (running) throw new StoreError(409, 'openre.destination_running', 'the destination is running; it has been asked to stop, delete it again in a few seconds');-- q.deleteDest.run(id);-- })();++ await q.deleteDest.run(id);++ });return true;}/** Destination row with its secrets opened, for the restream worker only. */-- function destinationForWorker(id) {-- const d = q.getDest.get(id);++ async function destinationForWorker(id) {++ const d = await q.getDest.get(id);if (!d) return null;return { ...d, stream_key: box.open(d.stream_key_enc), srt_passphrase: box.open(d.srt_passphrase_enc) };}-- function markDestinationFailure(id, error) {-- const d = q.getDest.get(id);++ async function markDestinationFailure(id, error) {++ const d = await q.getDest.get(id);if (!d) return null;const n = (d.consecutive_failures || 0) + 1;// Same escalation as Live: 15 min, 1 h, 6 h, then 24 h.const mins = n <= 1 ? 15 : n === 2 ? 60 : n === 3 ? 360 : 1440;-- db.prepare('UPDATE destinations SET consecutive_failures = ?, last_error = ?, last_failed_at = ?, cooldown_until = ?, updated_at = ? WHERE id = ?')++ await db.prepare('UPDATE destinations SET consecutive_failures = ?, last_error = ?, last_failed_at = ?, cooldown_until = ?, updated_at = ? WHERE id = ?').run(n, String(error || 'restream failed to go live').slice(0, 300), now(), now() + mins * 60000, now(), id);return { failures: n, cooldownMinutes: mins };}-- function clearDestinationCooldown(id) {-- db.prepare('UPDATE destinations SET consecutive_failures = 0, cooldown_until = NULL, last_error = NULL WHERE id = ? AND (consecutive_failures > 0 OR cooldown_until IS NOT NULL OR last_error IS NOT NULL)').run(id);++ async function clearDestinationCooldown(id) {++ await db.prepare('UPDATE destinations SET consecutive_failures = 0, cooldown_until = NULL, last_error = NULL WHERE id = ? AND (consecutive_failures > 0 OR cooldown_until IS NOT NULL OR last_error IS NOT NULL)').run(id);}// ── Outputs ───────────────────────────────────────────────@@@ -217,13 +217,13 @@ function createOutputs({ db, config, events, clock, box, definitions, sessions }}/** Coordinator: every live session gets an output per enabled, auto-start, usable destination. */-- function ensureAutoOutputs() {-- const rows = db.prepare(`SELECT s.id AS session_id, d.id AS destination_id FROM ingest_sessions s++ async function ensureAutoOutputs() {++ const rows = await db.prepare(`SELECT s.id AS session_id, d.id AS destination_id FROM ingest_sessions sJOIN destinations d ON d.definition_id = s.definition_idLEFT JOIN outputs o ON o.session_id = s.id AND o.destination_id = d.idWHERE s.state = 'live' AND d.enabled = 1 AND d.auto_start = 1 AND d.hold_reason IS NULLAND d.stream_key_enc IS NOT NULL AND (d.cooldown_until IS NULL OR d.cooldown_until <= ?) AND o.id IS NULL`).all(now());-- for (const r of rows) q.insertOut.run(newId('output', now()), r.session_id, r.destination_id, now(), now());++ for (const r of rows) await q.insertOut.run(newId('output', now()), r.session_id, r.destination_id, now(), now());return rows.length;}@@@ -231,44 +231,44 @@ function createOutputs({ db, config, events, clock, box, definitions, sessions }* Manual start (the owner pressed Start): creates or re-arms the output for the destination's* current live session. A manual start ignores a failure cooldown, like Live's.*/-- function startDestination(destinationId) {-- const d = q.getDest.get(destinationId);++ async function startDestination(destinationId) {++ const d = await q.getDest.get(destinationId);if (!d) throw new StoreError(404, 'openre.destination_not_found', 'no such destination');if (d.hold_reason) throw new StoreError(409, 'openre.destination_held', `held: ${d.hold_reason}`);if (!d.enabled) throw new StoreError(409, 'openre.destination_disabled', 'enable the destination first');if (!d.stream_key_enc) throw new StoreError(409, 'openre.destination_no_key', 'the destination has no stream key');-- const session = sessions.list({ definition_id: d.definition_id, state: 'live', limit: 1 })[0];++ const session = (await sessions.list({ definition_id: d.definition_id, state: 'live', limit: 1 }))[0];if (!session) throw new StoreError(409, 'openre.not_live', 'the stream is not live');-- return db.transaction(() => {-- clearDestinationCooldown(d.id);-- const existing = q.outBySessionDest.get(session.id, d.id);++ return await db.tx(async () => {++ await clearDestinationCooldown(d.id);++ const existing = await q.outBySessionDest.get(session.id, d.id);if (!existing) {const id = newId('output', now());-- q.insertOut.run(id, session.id, d.id, now(), now());-- return publicOutput(q.getOut.get(id));++ await q.insertOut.run(id, session.id, d.id, now(), now());++ return publicOutput(await q.getOut.get(id));}if (['pending', 'starting', 'live', 'error'].includes(existing.state) && existing.desired === 'run') return publicOutput(existing);if (['pending', 'starting', 'live', 'error'].includes(existing.state)) {-- db.prepare("UPDATE outputs SET desired = 'run', updated_at = ? WHERE id = ?").run(now(), existing.id);++ await db.prepare("UPDATE outputs SET desired = 'run', updated_at = ? WHERE id = ?").run(now(), existing.id);} else {-- db.prepare(`UPDATE outputs SET desired = 'run', state = 'pending', worker_id = NULL, worker_generation = NULL,++ await db.prepare(`UPDATE outputs SET desired = 'run', state = 'pending', worker_id = NULL, worker_generation = NULL,restart_attempts = 0, next_restart_at = NULL, last_error = NULL, ended_at = NULL, updated_at = ? WHERE id = ?`).run(now(), existing.id);}-- return publicOutput(q.getOut.get(existing.id));-- })();++ return publicOutput(await q.getOut.get(existing.id));++ });}-- function stopDestination(destinationId) {-- const n = db.prepare(`UPDATE outputs SET desired = 'stop', updated_at = ? WHERE destination_id = ? AND desired = 'run' AND state IN ${OPEN}`).run(now(), destinationId).changes;++ async function stopDestination(destinationId) {++ const n = (await db.prepare(`UPDATE outputs SET desired = 'stop', updated_at = ? WHERE destination_id = ? AND desired = 'run' AND state IN ${OPEN}`).run(now(), destinationId)).changes;// Pending outputs nobody has picked up can stop right here.-- db.prepare("UPDATE outputs SET state = 'stopped', ended_at = ?, updated_at = ? WHERE destination_id = ? AND desired = 'stop' AND state = 'pending' AND worker_id IS NULL").run(now(), now(), destinationId);++ await db.prepare("UPDATE outputs SET state = 'stopped', ended_at = ?, updated_at = ? WHERE destination_id = ? AND desired = 'stop' AND state = 'pending' AND worker_id IS NULL").run(now(), now(), destinationId);return n;}/** Coordinator: hand pending outputs to the newest ready restream generation. */-- function assignPending(worker) {++ async function assignPending(worker) {let n = 0;-- for (const o of q.pendingUnassigned.all()) n += q.assign.run(worker.id, worker.generation, now(), o.id).changes;++ for (const o of await q.pendingUnassigned.all()) n += (await q.assign.run(worker.id, worker.generation, now(), o.id)).changes;return n;}@@@ -278,9 +278,9 @@ function createOutputs({ db, config, events, clock, box, definitions, sessions }* is still assigned to that worker: a worker the coordinator has given up (lost) or that handed* the output over must not overwrite what the new owner does.*/-- function report(outputId, change, { workerId } = {}) {-- return db.transaction(() => {-- const o = q.getOut.get(outputId);++ async function report(outputId, change, { workerId } = {}) {++ return await db.tx(async () => {++ const o = await q.getOut.get(outputId);if (!o) return null;if (workerId && o.worker_id !== workerId) return null;const set = { updated_at: now() };@@@ -290,13 +290,13 @@ function createOutputs({ db, config, events, clock, box, definitions, sessions }if (change.progress !== undefined) set.progress = change.progress ? JSON.stringify(change.progress) : null;if (change.state === 'live') set.ever_live = 1;const cols = Object.keys(set);-- db.prepare(`UPDATE outputs SET ${cols.map(c => `${c} = @${c}`).join(', ')} WHERE id = @id`).run({ ...set, id: outputId });-- const after = q.getOut.get(outputId);++ await db.prepare(`UPDATE outputs SET ${cols.map(c => `${c} = @${c}`).join(', ')} WHERE id = @id`).run({ ...set, id: outputId });++ const after = await q.getOut.get(outputId);if (change.state && change.state !== o.state && (change.state === 'live' || change.state === 'failed')) {-- const dest = q.getDest.get(o.destination_id);-- const session = sessions.get(o.session_id);++ const dest = await q.getDest.get(o.destination_id);++ const session = await sessions.get(o.session_id);if (dest && session) {-- events.enqueue({++ await events.enqueue({event_type: change.state === 'live' ? TYPES.outputHealthy : TYPES.outputFailed,actor: { type: 'service', id: 'openre' },subject: { type: 'output', id: o.id, revision: after.restart_attempts + 1 },@@@ -315,43 +315,43 @@ function createOutputs({ db, config, events, clock, box, definitions, sessions }}}return after;-- })();++ });}-- function log(outputId, destinationId, level, message) {++ async function log(outputId, destinationId, level, message) {const text = String(message || '').slice(0, 500);-- q.insertLog.run(outputId || null, destinationId, level, text, now());-- q.pruneLogs.run(destinationId, outputId || null, outputId || null, config.outputs.logsPerOutput);++ await q.insertLog.run(outputId || null, destinationId, level, text, now());++ await q.pruneLogs.run(destinationId, outputId || null, outputId || null, config.outputs.logsPerOutput);}return {PLATFORMS, QUALITY_PRESETS, ENCODER_PRESETS,publicDest,publicOutput,-- getDestination: (id) => (isId('destination', id) ? publicDest(q.getDest.get(id)) : null),-- destinationRow: (id) => q.getDest.get(id),-- destinations: (definitionId) => q.destsOf.all(definitionId).map(publicDest),++ getDestination: async (id) => (isId('destination', id) ? publicDest(await q.getDest.get(id)) : null),++ destinationRow: async (id) => await q.getDest.get(id),++ destinations: async (definitionId) => (await q.destsOf.all(definitionId)).map(publicDest),createDestination,updateDestination,deleteDestination,destinationForWorker,markDestinationFailure,clearDestinationCooldown,-- getOutput: (id) => (isId('output', id) ? publicOutput(q.getOut.get(id)) : null),-- outputRow: (id) => q.getOut.get(id),-- outputsOfSession: (sessionId) => q.outsOfSession.all(sessionId).map(publicOutput),++ getOutput: async (id) => (isId('output', id) ? publicOutput(await q.getOut.get(id)) : null),++ outputRow: async (id) => await q.getOut.get(id),++ outputsOfSession: async (sessionId) => (await q.outsOfSession.all(sessionId)).map(publicOutput),ensureAutoOutputs,startDestination,stopDestination,assignPending,-- forWorker: (workerId) => q.forWorker.all(workerId),-- release: (workerId) => q.release.run(now(), workerId).changes,-- unassign: (outputId) => q.unassign.run(now(), outputId).changes,-- stopOrphans: () => q.stopOrphans.run(now(), now()).changes,++ forWorker: async (workerId) => await q.forWorker.all(workerId),++ release: async (workerId) => (await q.release.run(now(), workerId)).changes,++ unassign: async (outputId) => (await q.unassign.run(now(), outputId)).changes,++ stopOrphans: async () => (await q.stopOrphans.run(now(), now())).changes,report,log,-- logsOfOutput: (id, limit = 100) => q.logsOfOutput.all(id, limit),-- logsOfDestination: (id, limit = 100) => q.logsOfDest.all(id, limit),++ logsOfOutput: async (id, limit = 100) => await q.logsOfOutput.all(id, limit),++ logsOfDestination: async (id, limit = 100) => await q.logsOfDest.all(id, limit),};}@@@ -38,23 +38,23 @@ function createRecordings({ db, config, events, clock, definitions, sessions, loset: (cols) => db.prepare(`UPDATE recordings SET ${cols.map(c => `${c} = @${c}`).join(', ')}, updated_at = @now WHERE id = @id`),};-- function update(id, fields) {-- q.set(Object.keys(fields)).run({ ...fields, id, now: now() });-- return q.get.get(id);++ async function update(id, fields) {++ await q.set(Object.keys(fields)).run({ ...fields, id, now: now() });++ return await q.get.get(id);}-- function ensureRequests() {-- const rows = q.missing.all(now() - config.media.startDelayMs);-- for (const r of rows) q.insert.run(newId('recording', now()), r.session_id, r.mode, config.media.appId, now(), now());++ async function ensureRequests() {++ const rows = await q.missing.all(now() - config.media.startDelayMs);++ for (const r of rows) await q.insert.run(newId('recording', now()), r.session_id, r.mode, config.media.appId, now(), now());return rows.length;}-- function retryLater(rec, err, { disk = false } = {}) {++ async function retryLater(rec, err, { disk = false } = {}) {const attempts = rec.attempts + 1;const limit = disk ? DISK_MAX_ATTEMPTS : MAX_ATTEMPTS;-- if (attempts >= limit) return update(rec.id, { attempts, state: 'failed', last_error: String(err.message || err).slice(0, 300) });++ if (attempts >= limit) return await update(rec.id, { attempts, state: 'failed', last_error: String(err.message || err).slice(0, 300) });const wait = disk ? DISK_RETRY_MS : BACKOFF_MS[Math.min(attempts - 1, BACKOFF_MS.length - 1)];-- return update(rec.id, { attempts, next_attempt_at: now() + wait, last_error: String(err.message || err).slice(0, 300) });++ return await update(rec.id, { attempts, next_attempt_at: now() + wait, last_error: String(err.message || err).slice(0, 300) });}function refOf(definition, service, type) {@@@ -63,15 +63,15 @@ function createRecordings({ db, config, events, clock, definitions, sessions, lo}async function step(rec, media) {-- const session = sessions.get(rec.session_id);-- if (!session) return update(rec.id, { state: 'failed', last_error: 'session missing' });-- const definition = definitions.row(session.definition_id);++ const session = await sessions.get(rec.session_id);++ if (!session) return await update(rec.id, { state: 'failed', last_error: 'session missing' });++ const definition = await definitions.row(session.definition_id);const ended = isTerminal(session.state) || session.state === 'ending';if (rec.state === 'pending') {-- if (ended) return update(rec.id, { state: 'cancelled' });-- const pb = sessions.playback(session);-- if (!pb || !pb.rtmp) return retryLater(rec, new Error('no loopback play URL for this session yet'));++ if (ended) return await update(rec.id, { state: 'cancelled' });++ const pb = await sessions.playback(session);++ if (!pb || !pb.rtmp) return await retryLater(rec, new Error('no loopback play URL for this session yet'));let vodId = rec.media_vod_id;try {if (!vodId) {@@@ -88,7 +88,7 @@ function createRecordings({ db, config, events, clock, definitions, sessions, lometa: { source: 'openre', openre_session_id: session.id, openre_stream_id: definition.id, protocol: session.protocol, mode: rec.mode },});vodId = String(out && out.id);-- rec = update(rec.id, { media_vod_id: vodId, state: 'requested' });++ rec = await update(rec.id, { media_vod_id: vodId, state: 'requested' });}await media.ingestRtmp(vodId, pb.rtmp.internal_url);} catch (err) {@@@ -96,12 +96,12 @@ function createRecordings({ db, config, events, clock, definitions, sessions, lo// Media may have made the VOD row before the ingest was refused (disk low): drop the// empty shell so it never shows as a 0:00 VOD, and start over next time.if (vodId) await media.deleteVod(vodId).catch(() => {});-- update(rec.id, { media_vod_id: null, state: 'pending' });-- return retryLater(q.get.get(rec.id), err, { disk: /disk/i.test(String(err.message)) });++ await update(rec.id, { media_vod_id: null, state: 'pending' });++ return await retryLater(await q.get.get(rec.id), err, { disk: /disk/i.test(String(err.message)) });}-- return db.transaction(() => {-- const r = update(rec.id, { state: 'recording', attempts: 0, last_error: null, next_attempt_at: 0 });-- events.enqueue({++ return await db.tx(async () => {++ const r = await update(rec.id, { state: 'recording', attempts: 0, last_error: null, next_attempt_at: 0 });++ await events.enqueue({event_type: TYPES.recordingRequested,actor: { type: 'service', id: 'openre' },subject: { type: 'recording', id: rec.id, revision: 1 },@@@ -118,25 +118,25 @@ function createRecordings({ db, config, events, clock, definitions, sessions, lo},});return r;-- })();++ });}if (rec.state === 'requested') {// A create that succeeded and an ingest that did not: handled above; a requested row// here means the process died in between. Start the ingest again.-- return update(rec.id, { state: 'pending' });++ return await update(rec.id, { state: 'pending' });}-- if (rec.state === 'recording' && ended) rec = update(rec.id, { state: 'finalizing' });++ if (rec.state === 'recording' && ended) rec = await update(rec.id, { state: 'finalizing' });if (rec.state === 'finalizing') {try {await media.finalizeVod(rec.media_vod_id);if (rec.mode === 'clips') await media.deleteVod(rec.media_vod_id).catch(() => {});} catch (err) {// 409 = Media already finalising it (its ffmpeg saw the source end): that is success.-- if (err.status !== 409) return retryLater(rec, err);++ if (err.status !== 409) return await retryLater(rec, err);}-- return update(rec.id, { state: 'finalized', last_error: null });++ return await update(rec.id, { state: 'finalized', last_error: null });}return rec;}@@@ -145,7 +145,7 @@ function createRecordings({ db, config, events, clock, definitions, sessions, loasync function process(media) {if (!media.configured) return 0;let n = 0;-- for (const rec of q.due.all(now())) {++ for (const rec of await q.due.all(now())) {try { await step(rec, media); n++; } catch (err) { log.error(`[recording] ${rec.id}: ${err.stack || err}`); }}return n;@@@ -154,7 +154,7 @@ function createRecordings({ db, config, events, clock, definitions, sessions, loreturn {ensureRequests,process,-- bySession: (sessionId) => q.bySession.get(sessionId) || null,++ bySession: async (sessionId) => await q.bySession.get(sessionId) || null,};}@@@ -37,8 +37,8 @@ function createSessions({ db, config, events, clock, definitions, workers }) {};/** The Live channel this definition belongs to (the resolver's answer), or null while unresolved. */-- function liveLineage(definitionId) {-- const row = q.lineage.get(definitionId);++ async function liveLineage(definitionId) {++ const row = await q.lineage.get(definitionId);const r = row ? parseJson(row.resolution, null) : null;if (!r || r.status !== 'resolved' || !r.channel) return null;return { channel: { service: 'live', id: String(r.channel.id), slug: r.channel.slug, owner_subject: r.channel.owner_subject || null }, rule: r.rule || null };@@@ -49,11 +49,11 @@ function createSessions({ db, config, events, clock, definitions, workers }) {return { ...s, media_info: parseJson(s.media_info, null) };}-- function get(id) {-- return isId('session', id) ? shape(q.get.get(id)) : null;++ async function get(id) {++ return isId('session', id) ? shape(await q.get.get(id)) : null;}-- function list({ definition_id, owner_subject, state, limit = 50, before } = {}) {++ async function list({ definition_id, owner_subject, state, limit = 50, before } = {}) {const where = [];const args = [];if (definition_id) { where.push('s.definition_id = ?'); args.push(definition_id); }@@@ -61,13 +61,13 @@ function createSessions({ db, config, events, clock, definitions, workers }) {if (state === 'open') where.push(`s.state IN ${OPEN}`);else if (state) { where.push('s.state = ?'); args.push(state); }if (before) { where.push('s.created_at < ?'); args.push(Number(before)); }-- return db.prepare(`SELECT s.* FROM ingest_sessions s JOIN stream_definitions d ON d.id = s.definition_id++ return (await db.prepare(`SELECT s.* FROM ingest_sessions s JOIN stream_definitions d ON d.id = s.definition_id${where.length ? `WHERE ${where.join(' AND ')}` : ''} ORDER BY s.created_at DESC LIMIT ?`)-- .all(...args, Math.min(Math.max(Number(limit) || 50, 1), 200)).map(shape);++ .all(...args, Math.min(Math.max(Number(limit) || 50, 1), 200))).map(shape);}-- function envelopeFor(type, session, definition, extra = {}) {-- const worker = session.worker_id ? workers.get(session.worker_id) : null;++ async function envelopeFor(type, session, definition, extra = {}) {++ const worker = session.worker_id ? await workers.get(session.worker_id) : null;const payload = {session_id: session.id,stream_id: definition.id,@@@ -79,7 +79,7 @@ function createSessions({ db, config, events, clock, definitions, workers }) {worker: worker ? { id: worker.id, kind: worker.kind, generation: worker.generation } : null,external_refs: definition.external_refs,mirror_to_live: definition.mirror_to_live,-- lineage: liveLineage(definition.id),++ lineage: await liveLineage(definition.id),...extra,};return {@@@ -93,31 +93,31 @@ function createSessions({ db, config, events, clock, definitions, workers }) {}/**-- * Admit a publish on `worker` (synchronous; called inside the RTMP handshake).++ * Admit a publish on `worker` (called inside the RTMP handshake).* One open session per definition: a second publisher with a valid key is refused while the* first is alive, exactly like Live's in-process ingest. A leftover open session whose lease* has expired (its worker died) is failed first, so a crashed worker never locks a stream out.*/-- function admit({ definition, key, protocol, worker }) {-- return db.transaction(() => {-- for (const open of q.openForDefinition.all(definition.id)) {++ async function admit({ definition, key, protocol, worker }) {++ return await db.tx(async () => {++ for (const open of await q.openForDefinition.all(definition.id)) {const leaseOk = open.lease_expires_at && open.lease_expires_at > now();if (leaseOk) return { error: 'duplicate_publisher', existing: open.id };-- applyTransition(open, 'failed', { reason: 'lease_expired', actor: `worker:${worker.id}` });++ await applyTransition(open, 'failed', { reason: 'lease_expired', actor: `worker:${worker.id}` });}const id = newId('session', now());-- q.insert.run({++ await q.insert.run({id, definition_id: definition.id, key_id: key ? key.id : null, protocol,worker_id: worker.id, worker_kind: worker.kind, worker_generation: worker.generation,lease: now() + config.workers.leaseMs, now: now(),});-- q.transitionRow.run(id, null, 'starting', 'publish_accepted', `worker:${worker.id}`, now());-- return { session: shape(q.get.get(id)) };-- }).immediate();++ await q.transitionRow.run(id, null, 'starting', 'publish_accepted', `worker:${worker.id}`, now());++ return { session: shape(await q.get.get(id)) };++ });}// Inside a transaction.-- function applyTransition(current, to, { reason = null, actor = null, media_info } = {}) {++ async function applyTransition(current, to, { reason = null, actor = null, media_info } = {}) {if (!canTransition(current.state, to)) return { ok: false, code: 'invalid_transition', session: shape(current) };const t = now();const set = { state: to, updated_at: t, revision: current.revision + 1 };@@@ -128,21 +128,21 @@ function createSessions({ db, config, events, clock, definitions, workers }) {if (media_info !== undefined) set.media_info = media_info ? JSON.stringify(media_info) : null;if (isTerminal(to)) set.lease_expires_at = null;const cols = Object.keys(set);-- db.prepare(`UPDATE ingest_sessions SET ${cols.map(c => `${c} = @${c}`).join(', ')} WHERE id = @id AND state = @from`)++ await db.prepare(`UPDATE ingest_sessions SET ${cols.map(c => `${c} = @${c}`).join(', ')} WHERE id = @id AND state = @from`).run({ ...set, id: current.id, from: current.state });-- q.transitionRow.run(current.id, current.state, to, reason, actor, t);-- const session = shape(q.get.get(current.id));-- const definition = definitions.row(session.definition_id);++ await q.transitionRow.run(current.id, current.state, to, reason, actor, t);++ const session = shape(await q.get.get(current.id));++ const definition = await definitions.row(session.definition_id);if (to === 'live') {-- events.enqueue(envelopeFor(TYPES.sessionStarted, session, definition, { playback: playback(session) }));++ await events.enqueue(await envelopeFor(TYPES.sessionStarted, session, definition, { playback: await playback(session) }));} else if (to === 'ended' && session.live_at) {-- events.enqueue(envelopeFor(TYPES.sessionEnded, session, definition, {++ await events.enqueue(await envelopeFor(TYPES.sessionEnded, session, definition, {ended_at: iso(session.ended_at),duration_seconds: Math.round((session.ended_at - session.live_at) / 1000),end_reason: session.end_reason,}));} else if (to === 'failed') {-- events.enqueue(envelopeFor(TYPES.sessionFailed, session, definition, {++ await events.enqueue(await envelopeFor(TYPES.sessionFailed, session, definition, {ended_at: iso(session.ended_at),was_live: Boolean(session.live_at),duration_seconds: session.live_at ? Math.round((session.ended_at - session.live_at) / 1000) : 0,@@@ -151,45 +151,45 @@ function createSessions({ db, config, events, clock, definitions, workers }) {}if (isTerminal(to) || to === 'ending') {// Outputs follow their session down; they never pull the session with them.-- db.prepare("UPDATE outputs SET desired = 'stop', updated_at = ? WHERE session_id = ? AND desired = 'run'").run(t, session.id);++ await db.prepare("UPDATE outputs SET desired = 'stop', updated_at = ? WHERE session_id = ? AND desired = 'run'").run(t, session.id);}return { ok: true, session };}/** transition(id, to, { reason, actor, media_info }) -> { ok, session } | { ok: false, code } */-- function transition(id, to, opts = {}) {-- return db.transaction(() => {-- const current = q.get.get(id);++ async function transition(id, to, opts = {}) {++ return await db.tx(async () => {++ const current = await q.get.get(id);if (!current) return { ok: false, code: 'not_found' };-- return applyTransition(current, to, opts);-- }).immediate();++ return await applyTransition(current, to, opts);++ });}/** ending → ended in one call (the common clean stop). */-- function finish(id, { reason, actor } = {}) {-- return db.transaction(() => {-- let current = q.get.get(id);++ async function finish(id, { reason, actor } = {}) {++ return await db.tx(async () => {++ let current = await q.get.get(id);if (!current) return { ok: false, code: 'not_found' };if (current.state === 'starting' || current.state === 'live') {-- const r = applyTransition(current, 'ending', { reason, actor });++ const r = await applyTransition(current, 'ending', { reason, actor });if (!r.ok) return r;-- current = q.get.get(id);++ current = await q.get.get(id);}-- return applyTransition(current, 'ended', { reason, actor });-- }).immediate();++ return await applyTransition(current, 'ended', { reason, actor });++ });}-- function requestEnd(id, by) {-- return q.requestEnd.run(String(by || 'unknown'), now(), id).changes > 0;++ async function requestEnd(id, by) {++ return (await q.requestEnd.run(String(by || 'unknown'), now(), id)).changes > 0;}/*** Where to watch a session. Internal URLs are loopback addresses on the OpenRe host (Live and* Media run there too); public_url is served by openre-api's /play proxy.*/-- function playback(session) {++ async function playback(session) {if (!session) return null;-- const worker = session.worker_id ? workers.get(session.worker_id) : null;++ const worker = session.worker_id ? await workers.get(session.worker_id) : null;const ep = (worker && worker.endpoints) || {};const descriptor = {session_id: session.id,@@@ -224,13 +224,13 @@ function createSessions({ db, config, events, clock, definitions, workers }) {finish,requestEnd,playback,-- setMediaInfo: (id, info) => q.mediaInfo.run(JSON.stringify(info), id).changes > 0,-- transitions: (id) => q.transitions.all(id),-- endRequestsFor: (workerId) => q.endRequests.all(workerId).map(shape),-- ofWorker: (workerId) => q.ofWorker.all(workerId).map(shape),-- expiredLeases: () => q.expiredLeases.all(now()).map(shape),-- live: () => q.live.all().map(shape),-- openFor: (definitionId) => q.openForDefinition.all(definitionId).map(shape),++ setMediaInfo: async (id, info) => (await q.mediaInfo.run(JSON.stringify(info), id)).changes > 0,++ transitions: async (id) => await q.transitions.all(id),++ endRequestsFor: async (workerId) => (await q.endRequests.all(workerId)).map(shape),++ ofWorker: async (workerId) => (await q.ofWorker.all(workerId)).map(shape),++ expiredLeases: async () => (await q.expiredLeases.all(now())).map(shape),++ live: async () => (await q.live.all()).map(shape),++ openFor: async (definitionId) => (await q.openForDefinition.all(definitionId)).map(shape),};}@@@ -44,38 +44,38 @@ function createWorkers({ db, config, clock }) {const shape = (w) => (w ? { ...w, endpoints: parseJson(w.endpoints, {}) } : null);-- function register({ kind, release = config.release, pid = process.pid, host = os.hostname(), endpoints = {} }) {++ async function register({ kind, release = config.release, pid = process.pid, host = os.hostname(), endpoints = {} }) {if (!KINDS.includes(kind)) throw new Error(`unknown worker kind ${kind}`);const id = newId('worker', now());-- db.transaction(() => {-- const generation = q.maxGen.get(kind).g + 1;-- q.insert.run({ id, kind, generation, release, pid, host, endpoints: JSON.stringify(endpoints), now: now() });-- }).immediate();-- return shape(q.get.get(id));++ await db.tx(async () => {++ const generation = (await q.maxGen.get(kind)).g + 1;++ await q.insert.run({ id, kind, generation, release, pid, host, endpoints: JSON.stringify(endpoints), now: now() });++ });++ return shape(await q.get.get(id));}return {KINDS,register,-- get: (id) => shape(q.get.get(id)),-- ready: (id) => q.setReady.run(now(), now(), id).changes > 0,-- setEndpoints: (id, endpoints) => q.setEndpoints.run(JSON.stringify(endpoints), id),++ get: async (id) => shape(await q.get.get(id)),++ ready: async (id) => (await q.setReady.run(now(), now(), id)).changes > 0,++ setEndpoints: async (id, endpoints) => await q.setEndpoints.run(JSON.stringify(endpoints), id),/** Heartbeat + renew the leases of every session this worker owns, in one transaction. */-- heartbeat(id) {-- return db.transaction(() => {-- const alive = q.heartbeat.run(now(), id).changes > 0;-- if (alive) q.renewLeases.run(now() + config.workers.leaseMs, id);-- return shape(q.get.get(id));-- })();++ async heartbeat(id) {++ return await db.tx(async () => {++ const alive = (await q.heartbeat.run(now(), id)).changes > 0;++ if (alive) await q.renewLeases.run(now() + config.workers.leaseMs, id);++ return shape(await q.get.get(id));++ });},-- drain: (id, deadline) => q.drain.run(now(), deadline, id).changes > 0,-- stop: (id, reason) => q.stop.run(now(), reason || 'exit', id).changes > 0,-- lose: (id, reason) => q.lose.run(now(), reason || 'lease_expired', id).changes > 0,-- newestReady: (kind) => shape(q.newestReady.get(kind)),-- stale: (before) => q.stale.all(before).map(shape),-- olderReady: (kind, generation) => q.olderReady.all(kind, generation).map(shape),-- alive: () => q.alive.all().map(shape),-- recent: (limit = 50) => q.recent.all(limit).map(shape),++ drain: async (id, deadline) => (await q.drain.run(now(), deadline, id)).changes > 0,++ stop: async (id, reason) => (await q.stop.run(now(), reason || 'exit', id)).changes > 0,++ lose: async (id, reason) => (await q.lose.run(now(), reason || 'lease_expired', id)).changes > 0,++ newestReady: async (kind) => shape(await q.newestReady.get(kind)),++ stale: async (before) => (await q.stale.all(before)).map(shape),++ olderReady: async (kind, generation) => (await q.olderReady.all(kind, generation)).map(shape),++ alive: async () => (await q.alive.all()).map(shape),++ recent: async (limit = 50) => (await q.recent.all(limit)).map(shape),};}@@@ -53,8 +53,8 @@ function createUiRouter({ rt, auth }) {return false;}-- function ownDefinition(req, res, id) {-- const d = store.definitions.get(id);++ async function ownDefinition(req, res, id) {++ const d = await store.definitions.get(id);if (!d || d.state === 'archived' || !auth.canAccess(req.caller, d.owner_subject)) {page(req, res, { title: 'Not found', body: '<h1>Stream not found</h1><p><a href="/streams">Your streams</a></p>' }, 404);return null;@@@ -62,9 +62,9 @@ function createUiRouter({ rt, auth }) {return d;}-- function ownDestination(req, res, id) {-- const dest = store.outputs.destinationRow(id);-- const d = dest ? ownDefinition(req, res, dest.definition_id) : null;++ async function ownDestination(req, res, id) {++ const dest = await store.outputs.destinationRow(id);++ const d = dest ? await ownDefinition(req, res, dest.definition_id) : null;if (!dest && !res.headersSent) page(req, res, { title: 'Not found', body: '<h1>Destination not found</h1>' }, 404);return d ? { dest, definition: d } : null;}@@@ -83,33 +83,33 @@ function createUiRouter({ rt, auth }) {// What shipped on OpenRe.Stream: the shared update log every OpenVibe site has.router.get('/updates', (req, res) => page(req, res, { canonicalPath: '/updates', robots: 'index,follow', title: 'What shipped on OpenRe.Stream', body: frame.updatesBody({ service: 'openre', siteName: 'OpenRe.Stream' }) + `<script src="${ovServe.url('shipped.js')}" defer></script>` }));-- router.get('/', (req, res) => {++ router.get('/', async (req, res) => {const signedIn = req.caller.kind === 'user' && req.caller.subject;-- const mine = signedIn ? store.definitions.list({ owner_subject: req.caller.subject }) : [];++ const mine = signedIn ? await store.definitions.list({ owner_subject: req.caller.subject }) : [];page(req, res, {canonicalPath: '/', robots: 'index,follow',body: `<h1>OpenRe.Stream</h1><p>Ingest and restream for the OpenVibe network: stream definitions with hashed ingest keys, RTMP ingest sessions that run in transport workers separate from any web deploy, restream outputs with health and logs, and recording requests to OpenVibe.Media.</p><p class="muted">Status: alpha. RTMP ingest and RTMP/SRT restreaming work here; WHIP, WebRTC/SFU and JSMPEG are still served by OpenVibe.Live. Channels, discovery and watch pages stay on <a href="${esc(config.liveUrl)}">openvibe.live</a>.</p>--${signedIn ? `<h2>Your streams</h2>${mine.length ? streamTable(mine) : '<p class="muted">No streams yet.</p>'}<p><a href="/streams">Manage streams</a></p>` : '<p><a href="/auth/login?next=/streams">Sign in with OpenVibe</a> to manage your streams.</p>'}++${signedIn ? `<h2>Your streams</h2>${mine.length ? await streamTable(mine) : '<p class="muted">No streams yet.</p>'}<p><a href="/streams">Manage streams</a></p>` : '<p><a href="/auth/login?next=/streams">Sign in with OpenVibe</a> to manage your streams.</p>'}${frame.shipped({ service: 'openre', title: 'Recently shipped on OpenRe.Stream' })}`,});});-- function streamTable(list) {-- return `<table><tr><th>Stream</th><th>State</th><th>Now</th><th>Linked to</th></tr>${list.map((d) => {-- const open = store.sessions.openFor(d.id)[0];++ async function streamTable(list) {++ return `<table><tr><th>Stream</th><th>State</th><th>Now</th><th>Linked to</th></tr>${(await Promise.all(list.map(async (d) => {++ const open = (await store.sessions.openFor(d.id))[0];const refs = d.external_refs.map(r => `${r.service}:${r.type}:${r.id}`).join(', ');return `<tr><td><a href="/streams/${esc(d.id)}">${esc(d.title)}</a></td><td>${pill(d.state)}</td><td>${open ? pill(open.state) : '<span class="muted">offline</span>'}</td><td class="mono">${esc(refs) || '—'}</td></tr>`;-- }).join('')}</table>`;++ }))).join('')}</table>`;}-- router.get('/streams', (req, res) => {++ router.get('/streams', async (req, res) => {if (!needUser(req, res)) return;-- const list = store.definitions.list({ owner_subject: req.caller.subject });++ const list = await store.definitions.list({ owner_subject: req.caller.subject });page(req, res, {title: 'Streams', canonicalPath: '/streams',-- body: `<h1>Streams</h1>${list.length ? streamTable(list) : '<p class="muted">No streams yet.</p>'}++ body: `<h1>Streams</h1>${list.length ? await streamTable(list) : '<p class="muted">No streams yet.</p>'}<div class="card"><h2>New stream</h2>${form(req, '/streams', `<label>Title</label><input type="text" name="title" maxlength="140" required><div class="row"><div><label>Recording</label><select name="recording_mode"><option value="vod">Record a VOD</option><option value="clips">Clips only</option><option value="none">Do not record</option></select></div>@@@ -118,8 +118,8 @@ ${frame.shipped({ service: 'openre', title: 'Recently shipped on OpenRe.Stream'});});-- function keyPage(req, res, definition, key, heading) {-- const ep = store.definitions.ingestEndpoints(definition);++ async function keyPage(req, res, definition, key, heading) {++ const ep = await store.definitions.ingestEndpoints(definition);res.set('Cache-Control', 'no-store');page(req, res, {title: heading, canonicalPath: `/streams/${definition.id}`,@@@ -131,26 +131,26 @@ ${frame.shipped({ service: 'openre', title: 'Recently shipped on OpenRe.Stream'});}-- post('/streams', (req, res) => {++ post('/streams', async (req, res) => {const b = req.body || {};-- const { definition, key } = store.definitions.create({++ const { definition, key } = await store.definitions.create({owner_subject: req.caller.subject, title: b.title, recording_mode: b.recording_mode,recording_visibility: b.recording_visibility, created_by: req.caller.subject,});-- keyPage(req, res, definition, key, 'Stream created');++ await keyPage(req, res, definition, key, 'Stream created');});-- router.get('/streams/:id', (req, res) => {++ router.get('/streams/:id', async (req, res) => {if (!needUser(req, res)) return;-- const d = ownDefinition(req, res, req.params.id);++ const d = await ownDefinition(req, res, req.params.id);if (!d) return;-- const ep = store.definitions.ingestEndpoints(d);-- const keys = store.definitions.keys(d.id).filter(k => k.status !== 'revoked');-- const dests = store.outputs.destinations(d.id);-- const sessions = store.sessions.list({ definition_id: d.id, limit: 15 });-- const open = sessions.find(s => ['starting', 'live', 'ending'].includes(s.state));-- const outputs = open ? store.outputs.outputsOfSession(open.id) : [];-- const outFor = (destId) => outputs.find(o => o.destination_id === destId);++ const ep = await store.definitions.ingestEndpoints(d);++ const keys = (await store.definitions.keys(d.id)).filter(k => k.status !== 'revoked');++ const dests = await store.outputs.destinations(d.id);++ const sessions = await store.sessions.list({ definition_id: d.id, limit: 15 });++ const open = await sessions.find(s => ['starting', 'live', 'ending'].includes(s.state));++ const outputs = open ? await store.outputs.outputsOfSession(open.id) : [];++ const outFor = async (destId) => await outputs.find(o => o.destination_id === destId);page(req, res, {title: d.title, canonicalPath: `/streams/${d.id}`,body: `<h1>${esc(d.title)} ${pill(d.state)}</h1>@@@ -164,14 +164,14 @@ ${form(req, `/streams/${d.id}/rotate`, `<div class="row"><div><label>Old key sta<div class="card"><h2>Now</h2>${open ? `<p>${pill(open.state)} since ${esc(iso(open.live_at || open.created_at))} on ${esc(open.worker_kind)} generation ${esc(open.worker_generation)} · <a href="/sessions/${esc(open.id)}">session</a></p>${form(req, `/sessions/${open.id}/end`, '<button type="submit" class="danger">End session</button>', { cls: 'inline', confirm: 'Disconnect the encoder?' })}` : '<p class="muted">Offline.</p>'}</div><div class="card"><h2>Restream destinations</h2>--${dests.length ? `<table><tr><th>Destination</th><th>Output</th><th>Health</th><th>Actions</th></tr>${dests.map((x) => {-- const o = outFor(x.id);++${dests.length ? `<table><tr><th>Destination</th><th>Output</th><th>Health</th><th>Actions</th></tr>${(await Promise.all(dests.map(async (x) => {++ const o = await outFor(x.id);return `<tr><td><strong>${esc(x.name || x.platform)}</strong> <span class="muted">${esc(x.platform)}</span><br><span class="mono">${esc(x.server_url)}</span><br><span class="muted">key ${esc(x.stream_key_hint || 'not set')}${x.enabled ? '' : ' · disabled'}${x.auto_start ? ' · auto-start' : ''}${x.hold_reason ? ` · held: ${esc(x.hold_reason)}` : ''}${x.cooldown_ms ? ` · cooling down until ${esc(x.cooldown_until)}` : ''}</span></td><td>${o ? pill(o.state) : '<span class="muted">—</span>'}</td><td>${o && o.progress ? `${esc(o.progress.fps ?? '?')} fps · ${esc(o.progress.bitrate_kbps ?? '?')} kbit/s · speed ${esc(o.progress.speed ?? '?')}` : ''}${o && o.last_error ? `<br><span class="muted">${esc(o.last_error)}</span>` : ''}${!o && x.last_error ? `<span class="muted">${esc(x.last_error)}</span>` : ''}</td><td>${form(req, `/destinations/${x.id}/test`, '<button type="submit" class="secondary">Test</button>', { cls: 'inline' })}${open ? (o && ['pending', 'starting', 'live', 'error'].includes(o.state) ? form(req, `/destinations/${x.id}/stop`, '<button type="submit" class="secondary">Stop</button>', { cls: 'inline' }) : form(req, `/destinations/${x.id}/start`, '<button type="submit">Start</button>', { cls: 'inline' })) : ''}<a href="/destinations/${esc(x.id)}">Edit</a>${form(req, `/destinations/${x.id}/delete`, '<button type="submit" class="danger">Delete</button>', { cls: 'inline', confirm: 'Delete this destination?' })}</td></tr>`;-- }).join('')}</table>` : '<p class="muted">No destinations.</p>'}++ }))).join('')}</table>` : '<p class="muted">No destinations.</p>'}<h3>Add a destination</h3>${form(req, `/streams/${d.id}/destinations`, destinationFields({}))}</div><div class="card"><h2>Settings</h2>${form(req, `/streams/${d.id}`, `<label>Title</label><input type="text" name="title" maxlength="140" value="${esc(d.title)}">@@@ -215,35 +215,35 @@ ${dests.length ? `<table><tr><th>Destination</th><th>Output</th><th>Health</th><return `<table><tr><th>Session</th><th>State</th><th>Started</th><th>Ended</th><th>Worker</th></tr>${list.map(s => `<tr><td><a class="mono" href="/sessions/${esc(s.id)}">${esc(s.id)}</a></td><td>${pill(s.state)}${s.failure_reason ? ` <span class="muted">${esc(s.failure_reason)}</span>` : ''}</td><td>${esc(iso(s.live_at || s.created_at))}</td><td>${esc(iso(s.ended_at))}</td><td>${esc(s.worker_kind)} #${esc(s.worker_generation)}</td></tr>`).join('')}</table>`;}-- post('/streams/:id', (req, res) => {-- const d = ownDefinition(req, res, req.params.id);++ post('/streams/:id', async (req, res) => {++ const d = await ownDefinition(req, res, req.params.id);if (!d) return;const b = req.body || {};-- store.definitions.update(d.id, { title: b.title, recording_mode: b.recording_mode, recording_visibility: b.recording_visibility, playback_visibility: b.playback_visibility, state: b.state, mirror_to_live: b.mirror_to_live === '1' });++ await store.definitions.update(d.id, { title: b.title, recording_mode: b.recording_mode, recording_visibility: b.recording_visibility, playback_visibility: b.playback_visibility, state: b.state, mirror_to_live: b.mirror_to_live === '1' });res.redirect(303, `/streams/${d.id}`);});-- post('/streams/:id/rotate', (req, res) => {-- const d = ownDefinition(req, res, req.params.id);++ post('/streams/:id/rotate', async (req, res) => {++ const d = await ownDefinition(req, res, req.params.id);if (!d) return;-- const r = store.definitions.rotateKey(d.id, { grace_seconds: Number(req.body.grace_seconds) || 0, rotated_by: req.caller.subject });-- if (req.body.end_sessions === '1') for (const s of store.sessions.openFor(d.id)) store.sessions.requestEnd(s.id, `key_rotation:${req.caller.subject}`);-- keyPage(req, res, store.definitions.get(d.id), r.key, 'New ingest key');++ const r = await store.definitions.rotateKey(d.id, { grace_seconds: Number(req.body.grace_seconds) || 0, rotated_by: req.caller.subject });++ if (req.body.end_sessions === '1') for (const s of await store.sessions.openFor(d.id)) await store.sessions.requestEnd(s.id, `key_rotation:${req.caller.subject}`);++ await keyPage(req, res, await store.definitions.get(d.id), r.key, 'New ingest key');});-- post('/streams/:id/destinations', (req, res) => {-- const d = ownDefinition(req, res, req.params.id);++ post('/streams/:id/destinations', async (req, res) => {++ const d = await ownDefinition(req, res, req.params.id);if (!d) return;-- store.outputs.createDestination(d.id, destInput(req.body || {}, { editing: false }));++ await store.outputs.createDestination(d.id, destInput(req.body || {}, { editing: false }));res.redirect(303, `/streams/${d.id}`);});-- router.get('/destinations/:id', (req, res) => {++ router.get('/destinations/:id', async (req, res) => {if (!needUser(req, res)) return;-- const own = ownDestination(req, res, req.params.id);++ const own = await ownDestination(req, res, req.params.id);if (!own) return;const x = store.outputs.publicDest(own.dest);-- const logs = store.outputs.logsOfDestination(x.id, 50);++ const logs = await store.outputs.logsOfDestination(x.id, 50);page(req, res, {title: `Destination ${x.name || x.platform}`, canonicalPath: `/destinations/${x.id}`,body: `<h1>${esc(x.name || x.platform)}</h1><p><a href="/streams/${esc(own.definition.id)}">← ${esc(own.definition.title)}</a></p>@@@ -252,25 +252,25 @@ ${dests.length ? `<table><tr><th>Destination</th><th>Output</th><th>Health</th><});});-- post('/destinations/:id', (req, res) => {-- const own = ownDestination(req, res, req.params.id);++ post('/destinations/:id', async (req, res) => {++ const own = await ownDestination(req, res, req.params.id);if (!own) return;-- store.outputs.updateDestination(own.dest.id, destInput(req.body || {}, { editing: true }));++ await store.outputs.updateDestination(own.dest.id, destInput(req.body || {}, { editing: true }));res.redirect(303, `/streams/${own.definition.id}`);});-- post('/destinations/:id/delete', (req, res) => {-- const own = ownDestination(req, res, req.params.id);++ post('/destinations/:id/delete', async (req, res) => {++ const own = await ownDestination(req, res, req.params.id);if (!own) return;-- store.outputs.deleteDestination(own.dest.id);++ await store.outputs.deleteDestination(own.dest.id);res.redirect(303, `/streams/${own.definition.id}`);});post('/destinations/:id/test', async (req, res) => {-- const own = ownDestination(req, res, req.params.id);++ const own = await ownDestination(req, res, req.params.id);if (!own) return;const result = await testDestination(own.dest, { allowPrivate: config.outputs.allowPrivateHosts });-- store.outputs.log(null, own.dest.id, result.ok ? 'info' : 'warn', `test ${result.ok ? 'passed' : 'failed'}: ${result.checks.map(c => `${c.check} ${c.ok ? 'ok' : 'FAIL'} (${c.detail})`).join('; ')}`);++ await store.outputs.log(null, own.dest.id, result.ok ? 'info' : 'warn', `test ${result.ok ? 'passed' : 'failed'}: ${result.checks.map(c => `${c.check} ${c.ok ? 'ok' : 'FAIL'} (${c.detail})`).join('; ')}`);page(req, res, {title: 'Destination test',body: `<h1>Test ${result.ok ? pill('ok') : pill('failed')}</h1><table>${result.checks.map(c => `<tr><td>${esc(c.check)}</td><td>${c.ok ? pill('ok') : pill('failed')}</td><td>${esc(c.detail)}</td></tr>`).join('')}</table>@@@ -278,34 +278,34 @@ ${dests.length ? `<table><tr><th>Destination</th><th>Output</th><th>Health</th><});});-- post('/destinations/:id/start', (req, res) => {-- const own = ownDestination(req, res, req.params.id);++ post('/destinations/:id/start', async (req, res) => {++ const own = await ownDestination(req, res, req.params.id);if (!own) return;-- store.outputs.startDestination(own.dest.id);++ await store.outputs.startDestination(own.dest.id);res.redirect(303, `/streams/${own.definition.id}`);});-- post('/destinations/:id/stop', (req, res) => {-- const own = ownDestination(req, res, req.params.id);++ post('/destinations/:id/stop', async (req, res) => {++ const own = await ownDestination(req, res, req.params.id);if (!own) return;-- store.outputs.stopDestination(own.dest.id);++ await store.outputs.stopDestination(own.dest.id);res.redirect(303, `/streams/${own.definition.id}`);});-- router.get('/sessions', (req, res) => {++ router.get('/sessions', async (req, res) => {if (!needUser(req, res)) return;const all = req.caller.staff && req.query.all === '1';-- page(req, res, { title: 'Sessions', canonicalPath: '/sessions', body: `<h1>Sessions</h1>${req.caller.staff ? `<p class="muted"><a href="/sessions${all ? '' : '?all=1'}">${all ? 'Only mine' : 'Everyone (staff)'}</a></p>` : ''}${sessionTable(store.sessions.list({ owner_subject: all ? undefined : req.caller.subject, limit: 100 }))}` });++ page(req, res, { title: 'Sessions', canonicalPath: '/sessions', body: `<h1>Sessions</h1>${req.caller.staff ? `<p class="muted"><a href="/sessions${all ? '' : '?all=1'}">${all ? 'Only mine' : 'Everyone (staff)'}</a></p>` : ''}${sessionTable(await store.sessions.list({ owner_subject: all ? undefined : req.caller.subject, limit: 100 }))}` });});-- router.get('/sessions/:id', (req, res) => {++ router.get('/sessions/:id', async (req, res) => {if (!needUser(req, res)) return;-- const s = store.sessions.get(req.params.id);-- const d = s ? store.definitions.row(s.definition_id) : null;++ const s = await store.sessions.get(req.params.id);++ const d = s ? await store.definitions.row(s.definition_id) : null;if (!s || !d || !auth.canAccess(req.caller, d.owner_subject)) return page(req, res, { title: 'Not found', body: '<h1>Session not found</h1>' }, 404);-- const outputs = store.outputs.outputsOfSession(s.id);-- const rec = store.recordings.bySession(s.id);-- const pb = store.sessions.playback(s);++ const outputs = await store.outputs.outputsOfSession(s.id);++ const rec = await store.recordings.bySession(s.id);++ const pb = await store.sessions.playback(s);const open = ['starting', 'live', 'ending'].includes(s.state);return page(req, res, {title: `Session ${s.id}`, canonicalPath: `/sessions/${s.id}`,@@@ -318,16 +318,16 @@ ${dests.length ? `<table><tr><th>Destination</th><th>Output</th><th>Health</th><<tr><th>Recording</th><td>${rec ? `${pill(rec.state)} ${rec.media_vod_id ? `Media ${esc(rec.media_app)} VOD ${esc(rec.media_vod_id)}` : ''} ${esc(rec.last_error || '')}` : '<span class="muted">none</span>'}</td></tr><tr><th>Playback</th><td>${pb && pb.flv && s.state === 'live' ? `<a class="mono" href="${esc(pb.flv.public_url)}">${esc(pb.flv.public_url)}</a> (HTTP-FLV)` : '—'}</td></tr></table>${open ? form(req, `/sessions/${s.id}/end`, '<button type="submit" class="danger">End session</button>', { confirm: 'Disconnect the encoder?' }) : ''}</div>--<h2>Outputs</h2>${outputs.length ? `<table><tr><th>Output</th><th>State</th><th>Health</th><th>Restarts</th></tr>${outputs.map(o => `<tr><td class="mono">${esc(o.id)}<br><span class="muted">${esc(o.destination_id)}</span></td><td>${pill(o.state)}</td><td>${o.progress ? `${esc(o.progress.fps ?? '?')} fps · ${esc(o.progress.bitrate_kbps ?? '?')} kbit/s` : ''} ${esc(o.last_error || '')}</td><td>${esc(o.restart_attempts)}/${esc(o.max_restart_attempts)}</td></tr>`).join('')}</table>` : '<p class="muted">No outputs.</p>'}--<h2>Lifecycle</h2><table>${store.sessions.transitions(s.id).map(t => `<tr><td>${esc(iso(t.at))}</td><td>${esc(t.from_state || '·')} → ${pill(t.to_state)}</td><td>${esc(t.reason || '')}</td><td class="mono muted">${esc(t.actor || '')}</td></tr>`).join('')}</table>`,++<h2>Outputs</h2>${outputs.length ? `<table><tr><th>Output</th><th>State</th><th>Health</th><th>Restarts</th></tr>${(await outputs.map(o => `<tr><td class="mono">${esc(o.id)}<br><span class="muted">${esc(o.destination_id)}</span></td><td>${pill(o.state)}</td><td>${o.progress ? `${esc(o.progress.fps ?? '?')} fps · ${esc(o.progress.bitrate_kbps ?? '?')} kbit/s` : ''} ${esc(o.last_error || '')}</td><td>${esc(o.restart_attempts)}/${esc(o.max_restart_attempts)}</td></tr>`)).join('')}</table>` : '<p class="muted">No outputs.</p>'}++<h2>Lifecycle</h2><table>${(await store.sessions.transitions(s.id)).map(t => `<tr><td>${esc(iso(t.at))}</td><td>${esc(t.from_state || '·')} → ${pill(t.to_state)}</td><td>${esc(t.reason || '')}</td><td class="mono muted">${esc(t.actor || '')}</td></tr>`).join('')}</table>`,});});-- post('/sessions/:id/end', (req, res) => {-- const s = store.sessions.get(req.params.id);-- const d = s ? store.definitions.row(s.definition_id) : null;++ post('/sessions/:id/end', async (req, res) => {++ const s = await store.sessions.get(req.params.id);++ const d = s ? await store.definitions.row(s.definition_id) : null;if (!s || !d || !auth.canAccess(req.caller, d.owner_subject)) return page(req, res, { title: 'Not found', body: '<h1>Session not found</h1>' }, 404);-- store.sessions.requestEnd(s.id, req.caller.subject);++ await store.sessions.requestEnd(s.id, req.caller.subject);return res.redirect(303, `/sessions/${s.id}`);});@@@ -5,10 +5,14 @@// by the worker; a new API process starts and serves the same live session.const assert = require('assert');const path = require('path');--const { tmpDir, testEnv, child, waitFor, request, userToken, freePort, sleep, suite, OWNER } = require('./helpers');++const { tmpDir, testEnv, child, waitFor, request, userToken, freePort, sleep, suite, OWNER, multiProcess } = require('./helpers');const { load } = require('../server/config');const { openRuntime } = require('../server/store');++if (!multiProcess()) {++ console.log('api-restart: skipped (OpenRe processes sharing one database need PostgreSQL: npm run test:pg)');++ process.exit(0);++}const t = suite('api-restart');const dir = tmpDir();let env;@@@ -26,9 +30,9 @@ async function startApi() {}t('setup: API process + fake worker process holding a live session', async () => {-- env = testEnv(dir, { PORT: String(await freePort()), OPENRE_WORKER_HEARTBEAT_MS: '200', OPENRE_WORKER_LEASE_MS: '1500' });-- const rt = openRuntime({ config: load(env), log: { log() {}, warn() {}, error() {} } });-- const { definition } = rt.store.definitions.create({ owner_subject: OWNER, title: 'Restart test' });++ env = await testEnv(dir, { PORT: String(await freePort()), OPENRE_WORKER_HEARTBEAT_MS: '200', OPENRE_WORKER_LEASE_MS: '1500' });++ const rt = await openRuntime({ config: load(env), log: { log() {}, warn() {}, error() {} } });++ const { definition } = await rt.store.definitions.create({ owner_subject: OWNER, title: 'Restart test' });rt.db.close();api = await startApi();fake = child(path.join('test', 'fixtures', 'fake-worker.js'), { ...env, FAKE_STREAM_ID: definition.id });@@@ -39,18 +43,18 @@ t('setup: API process + fake worker process holding a live session', async () =>});t('stopping the API (SIGTERM) leaves the worker process and its session alone', async () => {-- const rt = openRuntime({ config: load(env), log: { log() {}, warn() {}, error() {} } });-- const leaseBefore = rt.store.sessions.get(sessionId).lease_expires_at;++ const rt = await openRuntime({ config: load(env), log: { log() {}, warn() {}, error() {} } });++ const leaseBefore = (await rt.store.sessions.get(sessionId)).lease_expires_at;api.proc.kill('SIGTERM');const exit = await api.proc.exited;assert.strictEqual(exit.code, 0, 'the API exits cleanly');// Longer than a lease: if anything depended on the API, the session would now be failing.await sleep(2500);assert.strictEqual(fake.exitCode, null, 'fake worker still running');-- const s = rt.store.sessions.get(sessionId);++ const s = await rt.store.sessions.get(sessionId);assert.strictEqual(s.state, 'live');assert.ok(s.lease_expires_at > leaseBefore, 'the worker kept renewing its lease while the API was down');-- assert.strictEqual(rt.store.workers.get(workerId).state, 'ready');++ assert.strictEqual((await rt.store.workers.get(workerId)).state, 'ready');rt.db.close();});@@@ -88,8 +88,8 @@ t('rotation over HTTP returns the new key once and the old one stops authenticatassert.strictEqual(r.status, 200);assert.notStrictEqual(r.body.key.key, firstKey);assert.strictEqual(r.body.retired[0].status, 'revoked');-- assert.strictEqual(api.rt.store.definitions.resolveIngestKey(firstKey, 'rtmp').error, 'revoked_key');-- assert.ok(api.rt.store.definitions.resolveIngestKey(r.body.key.key, 'rtmp').definition);++ assert.strictEqual((await api.rt.store.definitions.resolveIngestKey(firstKey, 'rtmp')).error, 'revoked_key');++ assert.ok((await api.rt.store.definitions.resolveIngestKey(r.body.key.key, 'rtmp')).definition);});t('destinations: validated, masked, never returned in full; SSRF rules apply', async () => {@@@ -129,13 +129,13 @@ t('the destination test reports URL, DNS and TCP checks without pushing media',t('sessions: list, detail with playback descriptor, end request', async () => {const rt = api.rt;-- const w = rt.store.workers.register({ kind: 'rtmp-ingest', endpoints: { publicPort: 1936, rtmpPlayPort: 19370, flvPort: 19371 } });-- rt.store.workers.ready(w.id);-- const def = rt.store.definitions.get(streamId);-- const a = rt.store.sessions.admit({ definition: def, key: null, protocol: 'rtmp', worker: rt.store.workers.get(w.id) });-- rt.store.sessions.transition(a.session.id, 'live');++ const w = await rt.store.workers.register({ kind: 'rtmp-ingest', endpoints: { publicPort: 1936, rtmpPlayPort: 19370, flvPort: 19371 } });++ await rt.store.workers.ready(w.id);++ const def = await rt.store.definitions.get(streamId);++ const a = await rt.store.sessions.admit({ definition: def, key: null, protocol: 'rtmp', worker: await rt.store.workers.get(w.id) });++ await rt.store.sessions.transition(a.session.id, 'live');const list = await request(api.base, 'GET', '/api/v1/sessions?state=live', { token: owner });-- assert.deepStrictEqual(list.body.sessions.map(s => s.id), [a.session.id]);++ assert.deepStrictEqual(await list.body.sessions.map(s => s.id), [a.session.id]);const detail = await request(api.base, 'GET', `/api/v1/sessions/${a.session.id}`, { token: owner });assert.strictEqual(detail.body.session.playback.flv.internal_url, `http://127.0.0.1:19371/live/${a.session.id}.flv`);assert.strictEqual(detail.body.session.playback.flv.public_url, `${api.config.baseUrl}/play/${a.session.id}.flv`);@@@ -145,11 +145,11 @@ t('sessions: list, detail with playback descriptor, end request', async () => {assert.strictEqual(noEndCap.status, 403);const end = await request(api.base, 'POST', `/api/v1/sessions/${a.session.id}/end`, { token: owner });assert.strictEqual(end.status, 202);-- assert.strictEqual(rt.store.sessions.get(a.session.id).desired_state, 'end');++ assert.strictEqual((await rt.store.sessions.get(a.session.id)).desired_state, 'end');// Archiving is refused while it is open.const del = await request(api.base, 'DELETE', `/api/v1/streams/${streamId}`, { token: owner });assert.strictEqual(del.status, 409);-- rt.store.sessions.finish(a.session.id, { reason: 'test' });++ await rt.store.sessions.finish(a.session.id, { reason: 'test' });assert.strictEqual((await request(api.base, 'GET', `/play/${a.session.id}.flv`)).status, 404, 'no playback for an ended session');});@@@ -9,8 +9,12 @@ const { load } = require('../server/config');const { openRuntime } = require('../server/store');const { createCoordinator } = require('../server/coordinator');const { localAddresses, rtmpProbe } = require('../scripts/cutover-preflight');--const { tmpDir, testEnv, child, waitFor, freePort, suite, silent } = require('./helpers');++const { tmpDir, testEnv, child, waitFor, freePort, suite, silent, multiProcess } = require('./helpers');++if (!multiProcess()) {++ console.log('bind-switch: skipped (OpenRe processes sharing one database need PostgreSQL: npm run test:pg)');++ process.exit(0);++}const t = suite('bind-switch');const procs = [];process.on('exit', () => { for (const p of procs) { try { p.kill('SIGKILL'); } catch { /* gone */ } } });@@@ -33,7 +37,7 @@ function startWorker(bind) {t('generation 1 on 127.0.0.1 only: the host\'s other addresses are refused', async () => {port = await freePort();const internal = await freePort();-- env = testEnv(tmpDir(), {++ env = await testEnv(tmpDir(), {OPENRE_RTMP_PORT: String(port),OPENRE_RTMP_INTERNAL_PORT_MIN: String(internal),OPENRE_RTMP_INTERNAL_PORT_MAX: String(internal + 20),@@@ -57,14 +61,14 @@ t('generation 2 starts with OPENRE_RTMP_BIND=0.0.0.0 on the same port while gene});t('the coordinator drains generation 1; it closes its listener and exits 0 by itself', async () => {-- rt = openRuntime({ config: load(env), log: silent });++ rt = await openRuntime({ config: load(env), log: silent });const coordinator = createCoordinator({ rt, media: null, log: silent });-- const out = coordinator.syncTick();++ const out = await coordinator.syncTick();assert.strictEqual(out.drained, 1);const r = await g1.exited;assert.strictEqual(r.code, 0, 'a drained, idle generation exits cleanly');assert.match(g1.output, /public listener closed/);-- const rows = rt.db.prepare("SELECT generation, state, stop_reason FROM workers WHERE kind = 'rtmp-ingest' ORDER BY generation").all();++ const rows = await rt.db.prepare("SELECT generation, state, stop_reason FROM workers WHERE kind = 'rtmp-ingest' ORDER BY generation").all();assert.deepStrictEqual(rows.map((w) => [w.generation, w.state]), [[1, 'stopped'], [2, 'ready']]);});@@@ -8,130 +8,130 @@ const { createWorkerRuntime } = require('../workers/runtime');const t = suite('coordinator');--function setup() {++async function setup() {const clock = manualClock();-- const rt = runtime({ clock });++ const rt = await runtime({ clock });const coordinator = createCoordinator({ rt, media: null, log: silent });-- const { definition, key } = rt.store.definitions.create({ owner_subject: OWNER });-- const worker = (kind) => { const w = rt.store.workers.register({ kind, endpoints: { flvPort: 19361, rtmpPlayPort: 19360 } }); rt.store.workers.ready(w.id); return rt.store.workers.get(w.id); };++ const { definition, key } = await rt.store.definitions.create({ owner_subject: OWNER });++ const worker = async (kind) => { const w = await rt.store.workers.register({ kind, endpoints: { flvPort: 19361, rtmpPlayPort: 19360 } }); await rt.store.workers.ready(w.id); return await rt.store.workers.get(w.id); };return { rt, clock, coordinator, definition, key, worker };}--t('generations grow per kind; when a newer one is ready every older one drains with a deadline', () => {-- const { rt, clock, coordinator, worker } = setup();-- const g1 = worker('rtmp-ingest');-- const r1 = worker('restream');++t('generations grow per kind; when a newer one is ready every older one drains with a deadline', async () => {++ const { rt, clock, coordinator, worker } = await setup();++ const g1 = await worker('rtmp-ingest');++ const r1 = await worker('restream');assert.strictEqual(g1.generation, 1);assert.strictEqual(r1.generation, 1);-- assert.strictEqual(coordinator.syncTick().drained, 0);-- const g2 = worker('rtmp-ingest');++ assert.strictEqual((await coordinator.syncTick()).drained, 0);++ const g2 = await worker('rtmp-ingest');assert.strictEqual(g2.generation, 2);-- assert.strictEqual(coordinator.syncTick().drained, 1);-- const old = rt.store.workers.get(g1.id);++ assert.strictEqual((await coordinator.syncTick()).drained, 1);++ const old = await rt.store.workers.get(g1.id);assert.strictEqual(old.state, 'draining');assert.strictEqual(old.drain_deadline, clock.now() + rt.config.workers.drainMaxMs);-- assert.strictEqual(rt.store.workers.get(r1.id).state, 'ready', 'other kinds are independent');-- assert.strictEqual(rt.store.workers.newestReady('rtmp-ingest').id, g2.id);++ assert.strictEqual((await rt.store.workers.get(r1.id)).state, 'ready', 'other kinds are independent');++ assert.strictEqual((await rt.store.workers.newestReady('rtmp-ingest')).id, g2.id);});--t('a draining generation keeps its live session; new sessions only reach the newest generation', () => {-- const { rt, coordinator, definition, key, worker } = setup();-- const g1 = worker('rtmp-ingest');-- const a = rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker: g1 });-- rt.store.sessions.transition(a.session.id, 'live');-- worker('rtmp-ingest');-- coordinator.syncTick();-- assert.strictEqual(rt.store.sessions.get(a.session.id).state, 'live');-- assert.strictEqual(rt.store.workers.get(g1.id).state, 'draining');++t('a draining generation keeps its live session; new sessions only reach the newest generation', async () => {++ const { rt, coordinator, definition, key, worker } = await setup();++ const g1 = await worker('rtmp-ingest');++ const a = await rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker: g1 });++ await rt.store.sessions.transition(a.session.id, 'live');++ await worker('rtmp-ingest');++ await coordinator.syncTick();++ assert.strictEqual((await rt.store.sessions.get(a.session.id)).state, 'live');++ assert.strictEqual((await rt.store.workers.get(g1.id)).state, 'draining');// The worker runtime of the draining generation refuses to take sessions (rtmp-ingest checks// runtime.draining / state === 'ready' in its publish handler).const exits = [];const wr = createWorkerRuntime({ rt, kind: 'rtmp-ingest', log: silent, exit: (c) => exits.push(c), hooks: { activeCount: () => 1 } });-- wr.register({});-- wr.ready();++ await wr.register({});++ await wr.ready();wr.stop();});--t('a worker without heartbeats is lost: its sessions fail (session.failed) and its outputs go back to pending', () => {-- const { rt, clock, coordinator, definition, key, worker } = setup();-- const ing = worker('rtmp-ingest');-- const rs = worker('restream');-- rt.store.outputs.createDestination(definition.id, { platform: 'custom', server_url: 'rtmp://127.0.0.1:1/app', stream_key: 'abcd1' });-- const a = rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker: ing });-- rt.store.sessions.transition(a.session.id, 'live');-- const tick1 = coordinator.syncTick();++t('a worker without heartbeats is lost: its sessions fail (session.failed) and its outputs go back to pending', async () => {++ const { rt, clock, coordinator, definition, key, worker } = await setup();++ const ing = await worker('rtmp-ingest');++ const rs = await worker('restream');++ await rt.store.outputs.createDestination(definition.id, { platform: 'custom', server_url: 'rtmp://127.0.0.1:1/app', stream_key: 'abcd1' });++ const a = await rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker: ing });++ await rt.store.sessions.transition(a.session.id, 'live');++ const tick1 = await coordinator.syncTick();assert.strictEqual(tick1.outputsCreated, 1);assert.strictEqual(tick1.outputsAssigned, 1);-- const out = rt.store.outputs.outputsOfSession(a.session.id)[0];++ const out = (await rt.store.outputs.outputsOfSession(a.session.id))[0];assert.strictEqual(out.worker.id, rs.id);// The restream worker dies; the ingest keeps heartbeating.clock.advance(rt.config.workers.leaseMs + 1000);-- rt.store.workers.heartbeat(ing.id);-- const tick2 = coordinator.syncTick();++ await rt.store.workers.heartbeat(ing.id);++ const tick2 = await coordinator.syncTick();assert.strictEqual(tick2.lost, 1);-- assert.strictEqual(rt.store.workers.get(rs.id).state, 'lost');-- assert.strictEqual(rt.store.sessions.get(a.session.id).state, 'live', 'a lost restream worker never ends the source session');-- assert.strictEqual(rt.store.outputs.getOutput(out.id).state, 'pending');-- assert.strictEqual(rt.store.outputs.getOutput(out.id).worker, null);++ assert.strictEqual((await rt.store.workers.get(rs.id)).state, 'lost');++ assert.strictEqual((await rt.store.sessions.get(a.session.id)).state, 'live', 'a lost restream worker never ends the source session');++ assert.strictEqual((await rt.store.outputs.getOutput(out.id)).state, 'pending');++ assert.strictEqual((await rt.store.outputs.getOutput(out.id)).worker, null);// The lost worker waking up and stopping its ffmpeg must not overwrite the released output.-- assert.strictEqual(rt.store.outputs.report(out.id, { state: 'stopped' }, { workerId: rs.id }), null);-- assert.strictEqual(rt.store.outputs.getOutput(out.id).state, 'pending');++ assert.strictEqual(await rt.store.outputs.report(out.id, { state: 'stopped' }, { workerId: rs.id }), null);++ assert.strictEqual((await rt.store.outputs.getOutput(out.id)).state, 'pending');// A new restream generation picks it up.-- const rs2 = worker('restream');-- assert.strictEqual(coordinator.syncTick().outputsAssigned, 1);-- assert.strictEqual(rt.store.outputs.getOutput(out.id).worker.id, rs2.id);++ const rs2 = await worker('restream');++ assert.strictEqual((await coordinator.syncTick()).outputsAssigned, 1);++ assert.strictEqual((await rt.store.outputs.getOutput(out.id)).worker.id, rs2.id);// Now the ingest worker dies too: its session fails.clock.advance(rt.config.workers.leaseMs + 1000);-- rt.store.workers.heartbeat(rs2.id);-- coordinator.syncTick();-- assert.strictEqual(rt.store.sessions.get(a.session.id).state, 'failed');-- assert.strictEqual(rt.store.sessions.get(a.session.id).failure_reason, 'worker_lost');-- assert.ok(outboxTypes(rt.db).includes('openre.session.failed'));-- assert.strictEqual(rt.store.outputs.getOutput(out.id).desired, 'stop');++ await rt.store.workers.heartbeat(rs2.id);++ await coordinator.syncTick();++ assert.strictEqual((await rt.store.sessions.get(a.session.id)).state, 'failed');++ assert.strictEqual((await rt.store.sessions.get(a.session.id)).failure_reason, 'worker_lost');++ assert.ok((await outboxTypes(rt.db)).includes('openre.session.failed'));++ assert.strictEqual((await rt.store.outputs.getOutput(out.id)).desired, 'stop');});--t('outputs are only created for enabled, auto-start destinations with a key and no cooldown', () => {-- const { rt, clock, coordinator, definition, key, worker } = setup();-- const ing = worker('rtmp-ingest');-- worker('restream');-- rt.store.outputs.createDestination(definition.id, { platform: 'custom', server_url: 'rtmp://127.0.0.1:1/a', stream_key: 'k1111' });-- rt.store.outputs.createDestination(definition.id, { platform: 'custom', server_url: 'rtmp://127.0.0.1:1/b', stream_key: 'k2222', auto_start: false });-- rt.store.outputs.createDestination(definition.id, { platform: 'custom', server_url: 'rtmp://127.0.0.1:1/c', stream_key: 'k3333', enabled: false });-- const cooling = rt.store.outputs.createDestination(definition.id, { platform: 'custom', server_url: 'rtmp://127.0.0.1:1/d', stream_key: 'k4444' });-- rt.store.outputs.markDestinationFailure(cooling.id, 'dead');-- const a = rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker: ing });-- rt.store.sessions.transition(a.session.id, 'live');-- assert.strictEqual(coordinator.syncTick().outputsCreated, 1);++t('outputs are only created for enabled, auto-start destinations with a key and no cooldown', async () => {++ const { rt, clock, coordinator, definition, key, worker } = await setup();++ const ing = await worker('rtmp-ingest');++ await worker('restream');++ await rt.store.outputs.createDestination(definition.id, { platform: 'custom', server_url: 'rtmp://127.0.0.1:1/a', stream_key: 'k1111' });++ await rt.store.outputs.createDestination(definition.id, { platform: 'custom', server_url: 'rtmp://127.0.0.1:1/b', stream_key: 'k2222', auto_start: false });++ await rt.store.outputs.createDestination(definition.id, { platform: 'custom', server_url: 'rtmp://127.0.0.1:1/c', stream_key: 'k3333', enabled: false });++ const cooling = await rt.store.outputs.createDestination(definition.id, { platform: 'custom', server_url: 'rtmp://127.0.0.1:1/d', stream_key: 'k4444' });++ await rt.store.outputs.markDestinationFailure(cooling.id, 'dead');++ const a = await rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker: ing });++ await rt.store.sessions.transition(a.session.id, 'live');++ assert.strictEqual((await coordinator.syncTick()).outputsCreated, 1);// A manual start ignores the cooldown (like Live's /start).-- rt.store.outputs.startDestination(cooling.id);-- assert.strictEqual(rt.store.outputs.outputsOfSession(a.session.id).length, 2);++ await rt.store.outputs.startDestination(cooling.id);++ assert.strictEqual((await rt.store.outputs.outputsOfSession(a.session.id)).length, 2);clock.advance(1);});--t('a restream generation at its drain deadline hands its outputs to the newest generation', () => {-- const { rt, coordinator, definition, key, worker } = setup();-- const ing = worker('rtmp-ingest');-- const rs1 = worker('restream');-- rt.store.outputs.createDestination(definition.id, { platform: 'custom', server_url: 'rtmp://127.0.0.1:1/a', stream_key: 'k1111' });-- const a = rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker: ing });-- rt.store.sessions.transition(a.session.id, 'live');-- coordinator.syncTick();-- const out = rt.store.outputs.outputsOfSession(a.session.id)[0];-- rt.store.outputs.report(out.id, { state: 'live' });-- const rs2 = worker('restream');-- coordinator.syncTick();-- assert.strictEqual(rt.store.workers.get(rs1.id).state, 'draining');-- assert.strictEqual(rt.store.outputs.getOutput(out.id).worker.id, rs1.id, 'the running output stays on the draining generation');-- assert.strictEqual(rt.store.outputs.release(rs1.id), 1); // what restream-worker does at its deadline-- coordinator.syncTick();-- assert.strictEqual(rt.store.outputs.getOutput(out.id).worker.id, rs2.id);-- assert.strictEqual(rt.store.sessions.get(a.session.id).state, 'live');++t('a restream generation at its drain deadline hands its outputs to the newest generation', async () => {++ const { rt, coordinator, definition, key, worker } = await setup();++ const ing = await worker('rtmp-ingest');++ const rs1 = await worker('restream');++ await rt.store.outputs.createDestination(definition.id, { platform: 'custom', server_url: 'rtmp://127.0.0.1:1/a', stream_key: 'k1111' });++ const a = await rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker: ing });++ await rt.store.sessions.transition(a.session.id, 'live');++ await coordinator.syncTick();++ const out = (await rt.store.outputs.outputsOfSession(a.session.id))[0];++ await rt.store.outputs.report(out.id, { state: 'live' });++ const rs2 = await worker('restream');++ await coordinator.syncTick();++ assert.strictEqual((await rt.store.workers.get(rs1.id)).state, 'draining');++ assert.strictEqual((await rt.store.outputs.getOutput(out.id)).worker.id, rs1.id, 'the running output stays on the draining generation');++ assert.strictEqual(await rt.store.outputs.release(rs1.id), 1); // what restream-worker does at its deadline++ await coordinator.syncTick();++ assert.strictEqual((await rt.store.outputs.getOutput(out.id)).worker.id, rs2.id);++ assert.strictEqual((await rt.store.sessions.get(a.session.id)).state, 'live');});t('only one coordinator acts at a time', async () => {-- const { rt } = setup();++ const { rt } = await setup();const a = createCoordinator({ rt, media: null, log: silent, holder: 'a' });const b = createCoordinator({ rt, media: null, log: silent, holder: 'b' });assert.ok(!(await a.tick()).skipped);@@@ -141,29 +141,29 @@ t('only one coordinator acts at a time', async () => {});t('the worker runtime exits by itself when a draining generation is idle, and on being lost', async () => {-- const { rt, worker } = setup();++ const { rt, worker } = await setup();const exits = [];let active = 1;const wr = createWorkerRuntime({ rt, kind: 'restream', log: silent, exit: (c) => exits.push(c), hooks: { activeCount: () => active } });-- wr.register({});-- wr.ready();-- worker('restream');-- rt.store.workers.drain(wr.me.id, Date.now() + 60000);-- wr.beat();++ await wr.register({});++ await wr.ready();++ await worker('restream');++ await rt.store.workers.drain(wr.me.id, Date.now() + 60000);++ await wr.beat();assert.ok(wr.draining);assert.deepStrictEqual(exits, [], 'still carrying an output');active = 0;-- wr.beat();++ await wr.beat();await new Promise(r => setImmediate(r));assert.deepStrictEqual(exits, [0]);-- assert.strictEqual(rt.store.workers.get(wr.me.id).state, 'stopped');++ assert.strictEqual((await rt.store.workers.get(wr.me.id)).state, 'stopped');const exits2 = [];const wr2 = createWorkerRuntime({ rt, kind: 'rtmp-ingest', log: silent, exit: (c) => exits2.push(c), hooks: { activeCount: () => 1 } });-- wr2.register({});-- wr2.ready();-- rt.store.workers.lose(wr2.me.id, 'test');-- wr2.beat();++ await wr2.register({});++ await wr2.ready();++ await rt.store.workers.lose(wr2.me.id, 'test');++ await wr2.beat();await new Promise(r => setImmediate(r));assert.deepStrictEqual(exits2, [1]);});@@@ -72,8 +72,8 @@ t('forms need the CSRF token; creating a stream shows the key once', async () =>assert.strictEqual(created.status, 200);assert.strictEqual(created.headers.get('cache-control'), 'no-store');const key = /(ork_[A-Za-z0-9_-]{43})/.exec(created.text)[1];-- const def = api.rt.store.definitions.list({ owner_subject: OWNER })[0];-- assert.strictEqual(api.rt.store.definitions.resolveIngestKey(key, 'rtmp').definition.id, def.id);++ const def = (await api.rt.store.definitions.list({ owner_subject: OWNER }))[0];++ assert.strictEqual((await api.rt.store.definitions.resolveIngestKey(key, 'rtmp')).definition.id, def.id);const page = await request(api.base, 'GET', `/streams/${def.id}`, { cookie });assert.strictEqual(page.status, 200);assert.ok(!page.text.includes(key), 'the stream page never shows the key again');@@@ -91,7 +91,7 @@ t('forms need the CSRF token; creating a stream shows the key once', async () =>const rotated = await request(api.base, 'POST', `/streams/${def.id}/rotate`, { cookie, form: { grace_seconds: '0', _csrf: csrfFor(token) } });const key2 = /(ork_[A-Za-z0-9_-]{43})/.exec(rotated.text)[1];assert.notStrictEqual(key2, key);-- assert.strictEqual(api.rt.store.definitions.resolveIngestKey(key, 'rtmp').error, 'revoked_key');++ assert.strictEqual((await api.rt.store.definitions.resolveIngestKey(key, 'rtmp')).error, 'revoked_key');await api.close();});@@@ -13,7 +13,7 @@ const { createRtmpIngest } = require('../workers/rtmp-ingest');const { createRestreamWorker } = require('../workers/restream-worker');const { createUnportedTransport } = require('../workers/unported-transport');const { openDb } = require('../server/db');--const { runtime, bootApi, request, serviceToken, tmpDir, testEnv, child, suite, silent } = require('./helpers');++const { runtime, bootApi, request, serviceToken, tmpDir, testEnv, child, suite, silent, testDb } = require('./helpers');const t = suite('drill-mode');// What the production env file sets: with these, a non-drill process would publish and record.@@@ -26,10 +26,10 @@ t('OPENRE_DRILL parses like the other switches and is off by default', () => {for (const v of ['', '0', 'false', 'off']) assert.strictEqual(load({ OPENRE_DRILL: v }).drill, false, v);});--t('no event relay and no Media client in a drill, even with production credentials', () => {-- const on = load(testEnv(tmpDir(), PROD_LIKE));-- const off = load(testEnv(tmpDir(), { ...PROD_LIKE, OPENRE_DRILL: '1' }));-- const db = openDb(':memory:');++t('no event relay and no Media client in a drill, even with production credentials', async () => {++ const on = load(await testEnv(tmpDir(), PROD_LIKE));++ const off = load(await testEnv(tmpDir(), { ...PROD_LIKE, OPENRE_DRILL: '1' }));++ const db = (await testDb(tmpDir())).db;assert.strictEqual(createEvents({ config: on, db, log: silent }).configured, true, 'control: the same env publishes outside a drill');const ev = createEvents({ config: off, db, log: silent });assert.strictEqual(ev.configured, false);@@@ -39,21 +39,21 @@ t('no event relay and no Media client in a drill, even with production credentiadb.close();});--t('the coordinator and every transport worker refuse to be created in a drill', () => {-- const rt = runtime({ env: { OPENRE_DRILL: '1' } });++t('the coordinator and every transport worker refuse to be created in a drill', async () => {++ const rt = await runtime({ env: { OPENRE_DRILL: '1' } });const drill = (fn) => assert.throws(fn, (err) => err.code === 'OPENRE_DRILL' && /restore drill/.test(err.message));drill(() => createCoordinator({ rt, media: null, log: silent }));drill(() => createRtmpIngest({ rt, log: silent, exit: () => {} }));drill(() => createRestreamWorker({ rt, log: silent, exit: () => {} }));for (const kind of ['webrtc-ingest', 'sfu', 'jsmpeg']) drill(() => createUnportedTransport({ rt, kind, log: silent, exit: () => {} }));-- assert.strictEqual(rt.db.prepare('SELECT count(*) AS n FROM workers').get().n, 0, 'no generation registered');-- assert.strictEqual(rt.db.prepare('SELECT count(*) AS n FROM leases').get().n, 0, 'no coordinator lease taken');++ assert.strictEqual((await rt.db.prepare('SELECT count(*) AS n FROM workers').get()).n, 0, 'no generation registered');++ assert.strictEqual((await rt.db.prepare('SELECT count(*) AS n FROM leases').get()).n, 0, 'no coordinator lease taken');});t('the entry points exit before opening the database', async () => {for (const script of ['workers/coordinator.js', 'workers/rtmp-ingest.js', 'workers/restream-worker.js', 'workers/sfu.js']) {const dir = tmpDir();-- const p = child(script, testEnv(dir, { ...PROD_LIKE, OPENRE_DRILL: '1' }));++ const p = child(script, await testEnv(dir, { ...PROD_LIKE, OPENRE_DRILL: '1' }));const r = await p.exited;assert.strictEqual(r.code, 1, `${script} exit code`);assert.match(p.output, /does not run in a restore drill/, script);@@@ -73,7 +73,7 @@ t('openre-api in a drill: reads work and match a normal instance; writes, /play/assert.strictEqual(met.status, 200, 'loopback /metrics (Track O)');const health = await request(normal.base, 'GET', '/api/health');const list = await request(normal.base, 'GET', '/api/v1/streams', { token: reader });-- const outboxBefore = normal.rt.db.prepare('SELECT count(*) AS n FROM event_outbox').get().n;++ const outboxBefore = (await normal.rt.db.prepare('SELECT count(*) AS n FROM event_outbox').get()).n;await normal.close();const drill = await bootApi({ dir, env: { ...PROD_LIKE, OPENRE_DRILL: '1' } });@@@ -100,9 +100,9 @@ t('openre-api in a drill: reads work and match a normal instance; writes, /play/assert.strictEqual(play.status, 503);const login = await request(drill.base, 'GET', '/auth/login');assert.strictEqual(login.status, 503);-- const count = drill.rt.db.prepare('SELECT count(*) AS n FROM stream_definitions').get().n;++ const count = (await drill.rt.db.prepare('SELECT count(*) AS n FROM stream_definitions').get()).n;assert.strictEqual(count, 1, 'nothing was written');-- assert.strictEqual(drill.rt.db.prepare('SELECT count(*) AS n FROM event_outbox').get().n, outboxBefore, 'no event enqueued');++ assert.strictEqual((await drill.rt.db.prepare('SELECT count(*) AS n FROM event_outbox').get()).n, outboxBefore, 'no event enqueued');} finally {await drill.close();}@@@ -9,12 +9,14 @@ const { load } = require('../../server/config');const { openRuntime } = require('../../server/store');const { createWorkerRuntime } = require('../../workers/runtime');--const rt = openRuntime({ config: load(), log: { log() {}, warn() {}, error: console.error } });--const worker = createWorkerRuntime({ rt, kind: 'rtmp-ingest', hooks: { activeCount: () => 1 }, log: { log() {}, warn() {}, error: console.error } });--worker.register({ publicPort: 0, rtmpPlayPort: 0, flvPort: 0 });--worker.ready();--const definition = rt.store.definitions.get(process.env.FAKE_STREAM_ID);--const a = rt.store.sessions.admit({ definition, key: null, protocol: 'rtmp', worker: worker.me });--rt.store.sessions.transition(a.session.id, 'live', { reason: 'fake_media' });--console.log(`SESSION ${a.session.id} WORKER ${worker.me.id}`);--setInterval(() => {}, 1 << 30);++(async () => {++ const rt = await openRuntime({ config: load(), log: { log() {}, warn() {}, error: console.error } });++ const worker = createWorkerRuntime({ rt, kind: 'rtmp-ingest', hooks: { activeCount: () => 1 }, log: { log() {}, warn() {}, error: console.error } });++ await worker.register({ publicPort: 0, rtmpPlayPort: 0, flvPort: 0 });++ await worker.ready();++ const definition = await rt.store.definitions.get(process.env.FAKE_STREAM_ID);++ const a = await rt.store.sessions.admit({ definition, key: null, protocol: 'rtmp', worker: worker.me });++ await rt.store.sessions.transition(a.session.id, 'live', { reason: 'fake_media' });++ console.log(`SESSION ${a.session.id} WORKER ${worker.me.id}`);++ setInterval(() => {}, 1 << 30);++})().catch((err) => { console.error(err); process.exit(1); });@@@ -2,6 +2,10 @@/*** Shared test fixtures: a generated Network signing key and token minting, temp databases, a* manual clock, runtime/API boot helpers, child-process helpers and a tiny sequential runner.++ *++ * Databases (ADR-035): one per test directory, migrated. npm test: PGlite in memory, so one process (a test that++ * spawns OpenRe processes sharing it says it is skipped); npm run test:pg (OPENRE_TEST_STORE=pg): a schema of its own++ * on the PostgreSQL + PgBouncer containers, whose URLs every spawned process of the directory gets.*/const crypto = require('crypto');const fs = require('fs');@@@ -10,8 +14,10 @@ const os = require('os');const path = require('path');const { spawn, spawnSync } = require('child_process');const { serviceAuth } = require('openvibe-contracts');++const { createTestDb } = require('openvibe-sdk/testing');const { load } = require('../server/config');const { openRuntime } = require('../server/store');++const { MIGRATIONS } = require('../server/db');const ROOT = path.join(__dirname, '..');const ISSUER = 'https://openvibe.network';@@@ -53,10 +59,27 @@ function tmpDir() {return d;}--function testEnv(dir, extra = {}) {++const STORE = process.env.OPENRE_TEST_STORE === 'pg' ? 'pg' : 'pglite';++const databases = new Map();++++/** The migrated database of a test directory; every runtime, API and spawned process of the directory shares it. */++async function testDb(dir) {++ if (!databases.has(dir)) databases.set(dir, createTestDb({ migrations: MIGRATIONS, store: STORE, service: 'openre' }));++ return await databases.get(dir);++}++++/** Several OpenRe processes on one database need PostgreSQL (PGlite is one process). */++const multiProcess = () => STORE === 'pg';++++/** A handle the test keeps: a runtime's or an API's close() leaves it open for the next one on the directory. */++const keepOpen = (db) => new Proxy(db, { get: (t, k) => (k === 'close' ? async () => {} : typeof t[k] === 'function' ? t[k].bind(t) : t[k]) });++++async function testEnv(dir, extra = {}) {++ const t = await testDb(dir);return {NODE_ENV: 'test',PORT: '0',++ ...(t.url ? { DATABASE_URL: t.url, DATABASE_DIRECT_URL: t.directUrl } : { OPENRE_PGLITE_DIR: path.join(dir, 'pglite') }),OPENRE_DB_PATH: path.join(dir, 'openre.db'),OPENRE_SECRETS_KEY: SECRETS_KEY,OV_NETWORK_PUBLIC_KEY: publicKey,@@@ -67,15 +90,15 @@ function testEnv(dir, extra = {}) {}/** A runtime on a temp database (no network). */--function runtime({ env = {}, clock, dir = tmpDir(), fetchImpl } = {}) {-- const config = load(testEnv(dir, env));-- return { ...openRuntime({ config, clock, log: silent, fetchImpl }), dir };++async function runtime({ env = {}, clock, dir = tmpDir(), fetchImpl } = {}) {++ const config = load(await testEnv(dir, env));++ return { ...await openRuntime({ config, clock, log: silent, fetchImpl, db: keepOpen((await testDb(dir)).db) }), dir };}async function bootApi({ env = {}, dir = tmpDir(), clock, fetchImpl } = {}) {const { start } = require('../server/index');-- const config = load(testEnv(dir, env));-- const h = await start({ config, clock, log: silent, fetchImpl });++ const config = load(await testEnv(dir, env));++ const h = await start({ config, clock, log: silent, fetchImpl, db: keepOpen((await testDb(dir)).db) });const base = `http://127.0.0.1:${h.server.address().port}`;return { ...h, base, dir };}@@@ -173,14 +196,14 @@ function suite(name) {}/** The event types in the outbox, in order. */--function outboxTypes(db) {-- return db.prepare('SELECT envelope FROM event_outbox ORDER BY id').all().map(r => JSON.parse(r.envelope).event_type);++async function outboxTypes(db) {++ return (await db.prepare('SELECT envelope FROM event_outbox ORDER BY id').all()).map(r => (typeof r.envelope === 'string' ? JSON.parse(r.envelope) : r.envelope).event_type);}--function outboxEnvelopes(db) {-- return db.prepare('SELECT envelope FROM event_outbox ORDER BY id').all().map(r => JSON.parse(r.envelope));++async function outboxEnvelopes(db) {++ return (await db.prepare('SELECT envelope FROM event_outbox ORDER BY id').all()).map(r => (typeof r.envelope === 'string' ? JSON.parse(r.envelope) : r.envelope));}module.exports = {-- ROOT, ISSUER, privateKey, publicKey, SECRETS_KEY, OWNER, OTHER, silent, serviceToken, userToken, manualClock, tmpDir, testEnv,++ ROOT, ISSUER, privateKey, publicKey, SECRETS_KEY, OWNER, OTHER, silent, serviceToken, userToken, manualClock, tmpDir, testEnv, testDb, multiProcess, keepOpen,runtime, bootApi, request, freePort, child, waitFor, sleep, hasFfmpeg, publish, rtmpSink, suite, outboxTypes, outboxEnvelopes,};@@@ -8,38 +8,38 @@ const { createCoordinator } = require('../server/coordinator');const t = suite('keys');--t('a new definition issues one key; only its SHA-256 is stored', () => {-- const rt = runtime();-- const { definition, key } = rt.store.definitions.create({ owner_subject: OWNER, title: 'Main' });++t('a new definition issues one key; only its SHA-256 is stored', async () => {++ const rt = await runtime();++ const { definition, key } = await rt.store.definitions.create({ owner_subject: OWNER, title: 'Main' });assert.ok(isIngestKeyShape(key.key), 'ork_ + 43 base64url chars');-- const rows = rt.db.prepare('SELECT * FROM ingest_keys').all();++ const rows = await rt.db.prepare('SELECT * FROM ingest_keys').all();assert.strictEqual(rows.length, 1);assert.strictEqual(rows[0].key_hash, hashIngestKey(key.key));assert.strictEqual(rows[0].hint, key.key.slice(-4));// The plain key is nowhere in the database file's tables.-- const dump = JSON.stringify(rt.db.prepare("SELECT * FROM ingest_keys").all()) + JSON.stringify(rt.db.prepare('SELECT * FROM stream_definitions').all());++ const dump = JSON.stringify(await rt.db.prepare("SELECT * FROM ingest_keys").all()) + JSON.stringify(await rt.db.prepare('SELECT * FROM stream_definitions').all());assert.ok(!dump.includes(key.key));-- const r = rt.store.definitions.resolveIngestKey(key.key, 'rtmp');++ const r = await rt.store.definitions.resolveIngestKey(key.key, 'rtmp');assert.strictEqual(r.definition.id, definition.id);-- assert.ok(rt.db.prepare('SELECT last_used_at FROM ingest_keys').get().last_used_at > 0);++ assert.ok((await rt.db.prepare('SELECT last_used_at FROM ingest_keys').get()).last_used_at > 0);});--t('unknown, malformed and wrong-protocol keys are refused', () => {-- const rt = runtime();-- const { key } = rt.store.definitions.create({ owner_subject: OWNER });-- assert.strictEqual(rt.store.definitions.resolveIngestKey('ork_' + 'A'.repeat(43), 'rtmp').error, 'unknown_key');-- assert.strictEqual(rt.store.definitions.resolveIngestKey('a3f9c2', 'rtmp').error, 'malformed_key');-- assert.strictEqual(rt.store.definitions.resolveIngestKey(key.key, 'whip').error, 'protocol_not_allowed');++t('unknown, malformed and wrong-protocol keys are refused', async () => {++ const rt = await runtime();++ const { key } = await rt.store.definitions.create({ owner_subject: OWNER });++ assert.strictEqual((await rt.store.definitions.resolveIngestKey('ork_' + 'A'.repeat(43), 'rtmp')).error, 'unknown_key');++ assert.strictEqual((await rt.store.definitions.resolveIngestKey('a3f9c2', 'rtmp')).error, 'malformed_key');++ assert.strictEqual((await rt.store.definitions.resolveIngestKey(key.key, 'whip')).error, 'protocol_not_allowed');});--t('rotate revokes the old key immediately and emits openre.key.rotated without any key', () => {-- const rt = runtime();-- const { definition, key: first } = rt.store.definitions.create({ owner_subject: OWNER, external_refs: [{ service: 'live', type: 'managed_stream', id: '7' }] });-- const r = rt.store.definitions.rotateKey(definition.id, { rotated_by: OWNER });++t('rotate revokes the old key immediately and emits openre.key.rotated without any key', async () => {++ const rt = await runtime();++ const { definition, key: first } = await rt.store.definitions.create({ owner_subject: OWNER, external_refs: [{ service: 'live', type: 'managed_stream', id: '7' }] });++ const r = await rt.store.definitions.rotateKey(definition.id, { rotated_by: OWNER });assert.notStrictEqual(r.key.key, first.key);-- assert.strictEqual(rt.store.definitions.resolveIngestKey(first.key, 'rtmp').error, 'revoked_key');-- assert.strictEqual(rt.store.definitions.resolveIngestKey(r.key.key, 'rtmp').definition.id, definition.id);-- const ev = outboxEnvelopes(rt.db).find(e => e.event_type === 'openre.key.rotated');++ assert.strictEqual((await rt.store.definitions.resolveIngestKey(first.key, 'rtmp')).error, 'revoked_key');++ assert.strictEqual((await rt.store.definitions.resolveIngestKey(r.key.key, 'rtmp')).definition.id, definition.id);++ const ev = (await outboxEnvelopes(rt.db)).find(e => e.event_type === 'openre.key.rotated');assert.ok(ev);assert.deepStrictEqual(ev.subject, { type: 'stream', id: definition.id, revision: 1 });assert.deepStrictEqual(ev.actor, { type: 'user', id: OWNER });@@@ -50,48 +50,48 @@ t('rotate revokes the old key immediately and emits openre.key.rotated without at('rotate with a grace period keeps the old key for new publishes until the coordinator revokes it', async () => {const clock = manualClock();-- const rt = runtime({ clock });-- const { definition, key: first } = rt.store.definitions.create({ owner_subject: OWNER });-- const r = rt.store.definitions.rotateKey(definition.id, { grace_seconds: 600 });-- assert.strictEqual(rt.store.definitions.resolveIngestKey(first.key, 'rtmp').key.status, 'grace');-- assert.ok(rt.store.definitions.resolveIngestKey(r.key.key, 'rtmp').definition);++ const rt = await runtime({ clock });++ const { definition, key: first } = await rt.store.definitions.create({ owner_subject: OWNER });++ const r = await rt.store.definitions.rotateKey(definition.id, { grace_seconds: 600 });++ assert.strictEqual((await rt.store.definitions.resolveIngestKey(first.key, 'rtmp')).key.status, 'grace');++ assert.ok((await rt.store.definitions.resolveIngestKey(r.key.key, 'rtmp')).definition);clock.advance(601 * 1000);// Past the grace period the key no longer authenticates, even before the coordinator runs...-- assert.strictEqual(rt.store.definitions.resolveIngestKey(first.key, 'rtmp').error, 'revoked_key');++ assert.strictEqual((await rt.store.definitions.resolveIngestKey(first.key, 'rtmp')).error, 'revoked_key');// ...and the coordinator records the revocation.-- const out = createCoordinator({ rt, media: null, log: { log() {}, warn() {}, error() {} } }).syncTick();++ const out = await createCoordinator({ rt, media: null, log: { log() {}, warn() {}, error() {} } }).syncTick();assert.strictEqual(out.keysRevoked, 1);-- assert.strictEqual(rt.db.prepare("SELECT status FROM ingest_keys WHERE key_hash = ?").get(hashIngestKey(first.key)).status, 'revoked');++ assert.strictEqual((await rt.db.prepare("SELECT status FROM ingest_keys WHERE key_hash = ?").get(hashIngestKey(first.key))).status, 'revoked');});--t('archiving revokes every key and is refused while a session is open', () => {-- const rt = runtime();-- const { definition, key } = rt.store.definitions.create({ owner_subject: OWNER });-- const w = rt.store.workers.register({ kind: 'rtmp-ingest' });-- rt.store.workers.ready(w.id);-- const a = rt.store.sessions.admit({ definition, key: { id: 'k' }, protocol: 'rtmp', worker: rt.store.workers.get(w.id) });-- assert.throws(() => rt.store.definitions.archive(definition.id), /end the live session/);-- rt.store.sessions.finish(a.session.id, { reason: 'test' });-- rt.store.definitions.archive(definition.id);-- assert.strictEqual(rt.store.definitions.resolveIngestKey(key.key, 'rtmp').error, 'revoked_key');++t('archiving revokes every key and is refused while a session is open', async () => {++ const rt = await runtime();++ const { definition, key } = await rt.store.definitions.create({ owner_subject: OWNER });++ const w = await rt.store.workers.register({ kind: 'rtmp-ingest' });++ await rt.store.workers.ready(w.id);++ const a = await rt.store.sessions.admit({ definition, key: { id: 'k' }, protocol: 'rtmp', worker: await rt.store.workers.get(w.id) });++ await assert.rejects(async () => await rt.store.definitions.archive(definition.id), /end the live session/);++ await rt.store.sessions.finish(a.session.id, { reason: 'test' });++ await rt.store.definitions.archive(definition.id);++ assert.strictEqual((await rt.store.definitions.resolveIngestKey(key.key, 'rtmp')).error, 'revoked_key');});--t('external refs are unique across definitions', () => {-- const rt = runtime();-- rt.store.definitions.create({ owner_subject: OWNER, external_refs: [{ service: 'live', type: 'managed_stream', id: '9' }] });-- assert.throws(() => rt.store.definitions.create({ owner_subject: OWNER, external_refs: [{ service: 'live', type: 'managed_stream', id: '9' }] }), /already belongs/);-- assert.ok(rt.store.definitions.findByRef('live', 'managed_stream', '9'));++t('external refs are unique across definitions', async () => {++ const rt = await runtime();++ await rt.store.definitions.create({ owner_subject: OWNER, external_refs: [{ service: 'live', type: 'managed_stream', id: '9' }] });++ await assert.rejects(async () => await rt.store.definitions.create({ owner_subject: OWNER, external_refs: [{ service: 'live', type: 'managed_stream', id: '9' }] }), /already belongs/);++ assert.ok(await rt.store.definitions.findByRef('live', 'managed_stream', '9'));});--t('destination keys are sealed with AES-256-GCM and only the last four characters come back', () => {-- const rt = runtime();-- const { definition } = rt.store.definitions.create({ owner_subject: OWNER });-- const d = rt.store.outputs.createDestination(definition.id, { platform: 'twitch', server_url: 'rtmps://live.twitch.tv/app', stream_key: 'live_1234567890_secret' });++t('destination keys are sealed with AES-256-GCM and only the last four characters come back', async () => {++ const rt = await runtime();++ const { definition } = await rt.store.definitions.create({ owner_subject: OWNER });++ const d = await rt.store.outputs.createDestination(definition.id, { platform: 'twitch', server_url: 'rtmps://live.twitch.tv/app', stream_key: 'live_1234567890_secret' });assert.strictEqual(d.stream_key_hint, '****cret');assert.ok(!JSON.stringify(d).includes('live_1234567890_secret'));-- const row = rt.db.prepare('SELECT stream_key_enc FROM destinations').get();++ const row = await rt.db.prepare('SELECT stream_key_enc FROM destinations').get();assert.ok(row.stream_key_enc.startsWith('v1.') && !row.stream_key_enc.includes('secret'));-- assert.strictEqual(rt.store.outputs.destinationForWorker(d.id).stream_key, 'live_1234567890_secret');++ assert.strictEqual((await rt.store.outputs.destinationForWorker(d.id)).stream_key, 'live_1234567890_secret');// A second key can take over: the previous one still opens old rows.const box2 = createBox({ key: require('crypto').randomBytes(32).toString('hex'), previous: rt.config.secretsKey });assert.strictEqual(box2.open(row.stream_key_enc), 'live_1234567890_secret');@@@ -8,9 +8,9 @@ const { createLineage } = require('../server/lineage');const t = suite('lineage');--function setup(answer) {++async function setup(answer) {const clock = manualClock();-- const rt = runtime({ clock });++ const rt = await runtime({ clock });const asked = [];const fetchImpl = async (url, opts = {}) => {const u = new URL(url);@@@ -29,34 +29,34 @@ function setup(answer) {const RESOLVED = { status: 'resolved', channel: { id: '17', slug: 'goosely', owner_subject: OWNER }, rule: 'slot', confidence: 'exact' };t('a Live-linked definition is resolved with its owner and slot; an unlinked one is never asked about', async () => {-- const { rt, lineage, asked } = setup(RESOLVED);-- const linked = rt.store.definitions.create({ owner_subject: OWNER, mirror_to_live: true, external_refs: [{ service: 'live', type: 'managed_stream', id: '12' }] }).definition;-- rt.store.definitions.create({ owner_subject: OWNER, title: 'OpenRe only' });++ const { rt, lineage, asked } = await setup(RESOLVED);++ const linked = (await rt.store.definitions.create({ owner_subject: OWNER, mirror_to_live: true, external_refs: [{ service: 'live', type: 'managed_stream', id: '12' }] })).definition;++ await rt.store.definitions.create({ owner_subject: OWNER, title: 'OpenRe only' });const out = await lineage.refresh();assert.deepStrictEqual(out, { checked: 1, resolved: 1 });assert.deepStrictEqual(asked, [{ owner_subject: OWNER, slot_id: '12' }], 'owner subject and slot id, never a display name');-- const row = rt.db.prepare('SELECT resolution FROM definition_lineage WHERE definition_id = ?').get(linked.id);++ const row = await rt.db.prepare('SELECT resolution FROM definition_lineage WHERE definition_id = ?').get(linked.id);assert.strictEqual(JSON.parse(row.resolution).channel.slug, 'goosely');assert.deepStrictEqual(await lineage.refresh(), { checked: 0, resolved: 0 }, 'fresh answers are not asked again');});t('session events carry the resolved channel', async () => {-- const { rt, coordinator } = setup(RESOLVED);-- const { definition, key } = rt.store.definitions.create({ owner_subject: OWNER, mirror_to_live: true, external_refs: [{ service: 'live', type: 'managed_stream', id: '12' }] });++ const { rt, coordinator } = await setup(RESOLVED);++ const { definition, key } = await rt.store.definitions.create({ owner_subject: OWNER, mirror_to_live: true, external_refs: [{ service: 'live', type: 'managed_stream', id: '12' }] });await coordinator.tick();-- const w = rt.store.workers.register({ kind: 'rtmp-ingest', endpoints: { flvPort: 19361, rtmpPlayPort: 19360 } }); rt.store.workers.ready(w.id);-- const a = rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker: rt.store.workers.get(w.id) });-- rt.store.sessions.transition(a.session.id, 'live');-- const started = outboxEnvelopes(rt.db).find((e) => e.event_type === 'openre.session.started');++ const w = await rt.store.workers.register({ kind: 'rtmp-ingest', endpoints: { flvPort: 19361, rtmpPlayPort: 19360 } }); await rt.store.workers.ready(w.id);++ const a = await rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker: await rt.store.workers.get(w.id) });++ await rt.store.sessions.transition(a.session.id, 'live');++ const started = (await outboxEnvelopes(rt.db)).find((e) => e.event_type === 'openre.session.started');assert.deepStrictEqual(started.payload.lineage, { channel: { service: 'live', id: '17', slug: 'goosely', owner_subject: OWNER }, rule: 'slot' });});t('unresolved answers are kept but not carried; a stale answer is asked again; Live down waits for the next tick', async () => {let mode = 'unresolved';-- const { rt, clock, lineage } = setup((u) => (mode === 'down' ? { status: 503, ok: false, json: async () => ({}) } : { status: 200, ok: true, json: async () => (mode === 'unresolved' ? { status: 'unresolved', reason: 'no_match' } : RESOLVED) }));-- const { definition } = rt.store.definitions.create({ owner_subject: OWNER, mirror_to_live: true });++ const { rt, clock, lineage } = await setup((u) => (mode === 'down' ? { status: 503, ok: false, json: async () => ({}) } : { status: 200, ok: true, json: async () => (mode === 'unresolved' ? { status: 'unresolved', reason: 'no_match' } : RESOLVED) }));++ const { definition } = await rt.store.definitions.create({ owner_subject: OWNER, mirror_to_live: true });assert.deepStrictEqual(await lineage.refresh(), { checked: 1, resolved: 0 });-- assert.strictEqual(rt.store.sessions.liveLineage(definition.id), null, 'an unresolved answer is not carried');++ assert.strictEqual(await rt.store.sessions.liveLineage(definition.id), null, 'an unresolved answer is not carried');mode = 'down';clock.advance(16 * 60 * 1000);assert.deepStrictEqual(await lineage.refresh(), { checked: 0, resolved: 0 });@@@ -67,7 +67,7 @@ t('unresolved answers are kept but not carried; a stale answer is asked again; Lt('off without a service secret', async () => {const clock = manualClock();-- const rt = runtime({ clock });++ const rt = await runtime({ clock });const off = createLineage({ db: rt.db, config: { ...rt.config, oauth: { ...rt.config.oauth, clientSecret: '' } }, env: {}, fetchImpl: () => { throw new Error('no calls'); }, log: silent });assert.strictEqual(off.enabled, false);assert.deepStrictEqual(await off.refresh(), { skipped: 'off' });@@@ -45,13 +45,13 @@ function liveSnapshot(dir) {return file;}--t('dry run writes nothing', () => {++t('dry run writes nothing', async () => {const dir = tmpDir();-- const rt = runtime({ dir });++ const rt = await runtime({ dir });const live = new Database(liveSnapshot(dir), { readonly: true });-- const report = migrate({ liveDb: live, rt, apply: false });-- assert.strictEqual(rt.db.prepare('SELECT COUNT(*) AS n FROM stream_definitions').get().n, 0);-- assert.strictEqual(rt.db.prepare('SELECT COUNT(*) AS n FROM migration_map').get().n, 0);++ const report = await migrate({ liveDb: live, rt, apply: false });++ assert.strictEqual((await rt.db.prepare('SELECT COUNT(*) AS n FROM stream_definitions').get()).n, 0);++ assert.strictEqual((await rt.db.prepare('SELECT COUNT(*) AS n FROM migration_map').get()).n, 0);assert.ok(report.counts.imported > 0);assert.match(checklist(report), /dry run/);});@@@ -59,17 +59,17 @@ t('dry run writes nothing', () => {let dir;let rt;let liveFile;--t('apply: slots with a subject are imported with new keys; the rest are held or excluded with reasons', () => {++t('apply: slots with a subject are imported with new keys; the rest are held or excluded with reasons', async () => {dir = tmpDir();-- rt = runtime({ dir, env: { OPENRE_DEST_ALLOW_PRIVATE: '0' } });++ rt = await runtime({ dir, env: { OPENRE_DEST_ALLOW_PRIVATE: '0' } });liveFile = liveSnapshot(dir);const before = fs.readFileSync(liveFile);const live = new Database(liveFile, { readonly: true });-- const report = migrate({ liveDb: live, rt, apply: true });++ const report = await migrate({ liveDb: live, rt, apply: true });live.close();assert.ok(before.equals(fs.readFileSync(liveFile)), 'the Live snapshot is untouched');-- const map = Object.fromEntries(rt.db.prepare('SELECT source_type, source_id, status, reason, target_id FROM migration_map').all().map(r => [`${r.source_type}:${r.source_id}`, r]));++ const map = Object.fromEntries((await rt.db.prepare('SELECT source_type, source_id, status, reason, target_id FROM migration_map').all()).map(r => [`${r.source_type}:${r.source_id}`, r]));assert.strictEqual(map['managed_stream:10'].status, 'imported');assert.strictEqual(map['managed_stream:11'].status, 'held');assert.match(map['managed_stream:11'].reason, /no canonical subject/);@@@ -77,21 +77,21 @@ t('apply: slots with a subject are imported with new keys; the rest are held orassert.strictEqual(map['managed_stream:13'].status, 'imported');assert.strictEqual(map['managed_stream:14'].status, 'imported');-- const def10 = rt.store.definitions.get(map['managed_stream:10'].target_id);++ const def10 = await rt.store.definitions.get(map['managed_stream:10'].target_id);assert.strictEqual(def10.owner_subject, SUBJECT_A);assert.strictEqual(def10.recording_visibility, 'unlisted', 'from the channel default');assert.strictEqual(def10.recording_mode, 'vod');assert.deepStrictEqual(def10.external_refs.map(r => `${r.service}:${r.type}:${r.id}`), ['live:managed_stream:10', 'live:user:1']);-- assert.strictEqual(rt.store.definitions.get(map['managed_stream:13'].target_id).recording_mode, 'none', 'VOD off on the channel, clips off on the slot');-- assert.strictEqual(rt.store.definitions.get(map['managed_stream:14'].target_id).recording_mode, 'clips');-- assert.strictEqual(rt.store.definitions.get(map['managed_stream:14'].target_id).recording_visibility, 'private');++ assert.strictEqual((await rt.store.definitions.get(map['managed_stream:13'].target_id)).recording_mode, 'none', 'VOD off on the channel, clips off on the slot');++ assert.strictEqual((await rt.store.definitions.get(map['managed_stream:14'].target_id)).recording_mode, 'clips');++ assert.strictEqual((await rt.store.definitions.get(map['managed_stream:14'].target_id)).recording_visibility, 'private');// Old keys: none of them authenticates, and none is stored in any form.for (const old of ['a3f9c2leakedkey0000000000000000000000000', 'personalkey111']) {-- assert.strictEqual(rt.db.prepare('SELECT COUNT(*) AS n FROM ingest_keys WHERE key_hash = ?').get(hashIngestKey(old)).n, 0);-- assert.ok(rt.store.definitions.resolveIngestKey(old, 'rtmp').error);++ assert.strictEqual((await rt.db.prepare('SELECT COUNT(*) AS n FROM ingest_keys WHERE key_hash = ?').get(hashIngestKey(old))).n, 0);++ assert.ok((await rt.store.definitions.resolveIngestKey(old, 'rtmp')).error);}-- assert.strictEqual(rt.db.prepare("SELECT COUNT(*) AS n FROM ingest_keys WHERE status = 'active'").get().n, 3, 'one new key per imported slot');++ assert.strictEqual((await rt.db.prepare("SELECT COUNT(*) AS n FROM ingest_keys WHERE status = 'active'").get()).n, 3, 'one new key per imported slot');// Destinations: sealed, LAN one held, OAuth-linked flagged, unbound one held (two slots).assert.strictEqual(map['restream_destination:100'].status, 'imported');@@@ -100,13 +100,13 @@ t('apply: slots with a subject are imported with new keys; the rest are held orassert.match(map['restream_destination:102'].reason, /private/);assert.strictEqual(map['restream_destination:103'].status, 'held');assert.match(map['restream_destination:103'].reason, /several slots/);-- const d100 = rt.store.outputs.destinationForWorker(map['restream_destination:100'].target_id);++ const d100 = await rt.store.outputs.destinationForWorker(map['restream_destination:100'].target_id);assert.strictEqual(d100.stream_key, 'live_twitchsecret_1');-- assert.ok(!JSON.stringify(rt.db.prepare('SELECT * FROM destinations').all()).includes('live_twitchsecret_1'), 'sealed at rest');-- const d101 = rt.store.outputs.getDestination(map['restream_destination:101'].target_id);++ assert.ok(!JSON.stringify(await rt.db.prepare('SELECT * FROM destinations').all()).includes('live_twitchsecret_1'), 'sealed at rest');++ const d101 = await rt.store.outputs.getDestination(map['restream_destination:101'].target_id);assert.strictEqual(d101.auto_start, false);assert.strictEqual(d101.custom_video_bitrate, 4500);-- const d102 = rt.store.outputs.getDestination(map['restream_destination:102'].target_id);++ const d102 = await rt.store.outputs.getDestination(map['restream_destination:102'].target_id);assert.strictEqual(d102.enabled, false);assert.ok(d102.hold_reason);@@@ -118,32 +118,32 @@ t('apply: slots with a subject are imported with new keys; the rest are held orfor (const secret of ['a3f9c2leakedkey', 'live_twitchsecret_1', 'yt-secret-2222', 'personalkey111', 'kick-secret-44']) assert.ok(!md.includes(secret), `no ${secret} in the checklist`);});--t('re-running is idempotent; a held slot is imported once its user has a subject', () => {-- const counts = () => ({-- defs: rt.db.prepare('SELECT COUNT(*) AS n FROM stream_definitions').get().n,-- dests: rt.db.prepare('SELECT COUNT(*) AS n FROM destinations').get().n,-- keys: rt.db.prepare('SELECT COUNT(*) AS n FROM ingest_keys').get().n,++t('re-running is idempotent; a held slot is imported once its user has a subject', async () => {++ const counts = async () => ({++ defs: (await rt.db.prepare('SELECT COUNT(*) AS n FROM stream_definitions').get()).n,++ dests: (await rt.db.prepare('SELECT COUNT(*) AS n FROM destinations').get()).n,++ keys: (await rt.db.prepare('SELECT COUNT(*) AS n FROM ingest_keys').get()).n,});-- const before = counts();++ const before = await counts();let live = new Database(liveFile, { readonly: true });-- const again = migrate({ liveDb: live, rt, apply: true });++ const again = await migrate({ liveDb: live, rt, apply: true });live.close();-- assert.deepStrictEqual(counts(), before);++ assert.deepStrictEqual(await counts(), before);assert.strictEqual(again.counts.imported, 0);const w = new Database(liveFile);-- w.prepare("INSERT INTO linked_accounts (user_id, service, service_user_id, subject_id) VALUES (2, 'network', '60', 'usr_01J00000000000000000000D01')").run();++ await w.prepare("INSERT INTO linked_accounts (user_id, service, service_user_id, subject_id) VALUES (2, 'network', '60', 'usr_01J00000000000000000000D01')").run();w.close();live = new Database(liveFile, { readonly: true });-- migrate({ liveDb: live, rt, apply: true });++ await migrate({ liveDb: live, rt, apply: true });live.close();-- assert.strictEqual(counts().defs, before.defs + 1);-- assert.strictEqual(rt.db.prepare("SELECT status FROM migration_map WHERE source_type = 'managed_stream' AND source_id = '11'").get().status, 'imported');++ assert.strictEqual((await counts()).defs, before.defs + 1);++ assert.strictEqual((await rt.db.prepare("SELECT status FROM migration_map WHERE source_type = 'managed_stream' AND source_id = '11'").get()).status, 'imported');});--t('the CLI runs a dry run against a snapshot and prints the checklist', () => {++t('the CLI runs a dry run against a snapshot and prints the checklist', async () => {const d = tmpDir();const file = liveSnapshot(d);-- const env = testEnv(d);++ const env = await testEnv(d);const out = execFileSync(process.execPath, [path.join(ROOT, 'scripts', 'migrate-from-live.js'), '--live-db', file], {env: { ...process.env, ...env, OPENRE_ENV_FILE: '/nonexistent' }, cwd: ROOT, encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'],});@@@ -1,6 +1,7 @@'use strict';// scripts/cutover-preflight.js against a mocked host: files, commands, DNS, interfaces, the port--// probe and /api/ready are fakes; the three databases are real SQLite files opened read-only.++// probe and /api/ready are fakes; OpenRe's database is a real PostgreSQL one (PGlite) read in READ ONLY transactions,++// Events' and Live's are real SQLite files opened read-only (Events' production PostgreSQL has the same tables).// Covers the host as it is today (2026-09-23: everything the cutover still needs fails), a host// ready for the first slot, each failure on its own, the off-host port check, the probe URL, the// per-slot check, and that no secret value ever reaches the output.@@@ -9,7 +10,8 @@ const net = require('net');const path = require('path');const Database = require('better-sqlite3');const { openDb } = require('../server/db');--const { preflight, parseArgs, format, parseSsListeners, rtmpProbe, sameSecret, CHECKS } = require('../scripts/cutover-preflight');++const { load } = require('../server/config');++const { preflight, parseArgs, format, parseSsListeners, rtmpProbe, sameSecret, withDb, CHECKS } = require('../scripts/cutover-preflight');const { tmpDir, freePort, suite } = require('./helpers');const t = suite('preflight');@@@ -23,18 +25,18 @@ const LIVE_PID = 4242;// ── fixtures: three databases ──────────────────────────────────────────────--function makeDbs() {++async function makeDbs() {const dir = tmpDir();-- const openrePath = path.join(dir, 'openre.db');-- const o = openDb(openrePath);++ // OpenRe's database: PGlite in memory, reached as the target 'pglite:openre' (host() below hands it out).++ const openrePath = 'pglite:openre';++ const o = await openDb(load({ NODE_ENV: 'test' }), { memory: true });const now = Date.now();const worker = o.prepare(`INSERT INTO workers (id, kind, generation, release, state, endpoints, started_at, heartbeat_at) VALUES (?, ?, ?, ?, ?, '{}', ?, ?)`);-- worker.run('wrk_1', 'rtmp-ingest', 2, RELEASE, 'ready', now, now);-- worker.run('wrk_2', 'restream', 2, RELEASE, 'ready', now, now);-- o.prepare(`INSERT INTO stream_definitions (id, owner_subject, mirror_to_live, created_at, updated_at) VALUES ('str_1', 'usr_1', 1, ?, ?)`).run(now, now);-- o.prepare(`INSERT INTO external_refs (definition_id, service, type, ref_id, created_at) VALUES ('str_1', 'live', 'managed_stream', '12', ?)`).run(now);-- o.prepare(`INSERT INTO migration_map (source_system, source_type, source_id, target_type, target_id, status, imported_at) VALUES ('live', 'managed_stream', '12', 'stream', 'str_1', 'imported', ?)`).run(now);-- o.close();++ await worker.run('wrk_1', 'rtmp-ingest', 2, RELEASE, 'ready', now, now);++ await worker.run('wrk_2', 'restream', 2, RELEASE, 'ready', now, now);++ await o.prepare(`INSERT INTO stream_definitions (id, owner_subject, mirror_to_live, created_at, updated_at) VALUES ('str_1', 'usr_1', 1, ?, ?)`).run(now, now);++ await o.prepare(`INSERT INTO external_refs (definition_id, service, type, ref_id, created_at) VALUES ('str_1', 'live', 'managed_stream', '12', ?)`).run(now);++ await o.prepare(`INSERT INTO migration_map (source_system, source_type, source_id, target_type, target_id, status, imported_at) VALUES ('live', 'managed_stream', '12', 'stream', 'str_1', 'imported', ?)`).run(now);const eventsPath = path.join(dir, 'events.db');const e = new Database(eventsPath);@@@ -52,18 +54,18 @@ function makeDbs() {INSERT INTO managed_streams (id, user_id, slug, title, protocol, streaming_method) VALUES (12, 5, 'test', 'Test', 'rtmp', 'obs');INSERT INTO linked_accounts (user_id, service, service_user_id, subject_id) VALUES (5, 'network', '77', 'usr_01J0000000000000000000000A');`);l.close();-- return { dir, openrePath, eventsPath, livePath };++ return { dir, openre: o, openrePath, eventsPath, livePath };}--function addSubscription(dbs, { secret = EVENTS_SECRET, enabled = 1, id = 'sub_1' } = {}) {++async function addSubscription(dbs, { secret = EVENTS_SECRET, enabled = 1, id = 'sub_1' } = {}) {const e = new Database(dbs.eventsPath);-- e.prepare(`INSERT INTO subscriptions (id, consumer, topic_pattern, endpoint, secret, enabled, created_at, updated_at) VALUES (?, 'live', 'openre.session.*', 'http://127.0.0.1:3000/internal/openre-events', ?, ?, 0, 0)`).run(id, secret, enabled);++ await e.prepare(`INSERT INTO subscriptions (id, consumer, topic_pattern, endpoint, secret, enabled, created_at, updated_at) VALUES (?, 'live', 'openre.session.*', 'http://127.0.0.1:3000/internal/openre-events', ?, ?, 0, 0)`).run(id, secret, enabled);e.close();}--function addDelivery(dbs, status, n = 1) {++async function addDelivery(dbs, status, n = 1) {const e = new Database(dbs.eventsPath);-- for (let i = 0; i < n; i++) e.prepare("INSERT INTO deliveries (event_id, subscription_id, seq, priority, status, created_at, updated_at) VALUES (?, 'sub_1', ?, 1, ?, 0, 0)").run(`evt_${status}_${i}`, i, status);++ for (let i = 0; i < n; i++) await e.prepare("INSERT INTO deliveries (event_id, subscription_id, seq, priority, status, created_at, updated_at) VALUES (?, 'sub_1', ?, 1, ?, 0, 0)").run(`evt_${status}_${i}`, i, status);e.close();}@@@ -72,6 +74,9 @@ function addDelivery(dbs, status, n = 1) {const envText = (o) => Object.entries(o).filter(([, v]) => v != null).map(([k, v]) => `${k}=${v}`).join('\n') + '\n';/** A host that is ready for the first slot. Every field can be overridden per test. */++/** The fixture's PGlite handle, which the preflight's close() must not end (every check reads it again). */++const keepOpen = (db) => new Proxy(db, { get: (t, k) => (k === 'close' ? async () => {} : typeof t[k] === 'function' ? t[k].bind(t) : t[k]) });++function host(dbs, over = {}) {const h = {release: RELEASE,@@@ -124,7 +129,7 @@ function host(dbs, over = {}) {if (h.probeUrl) return { status: h.probeUrl.status || 200, ok: (h.probeUrl.status || 200) < 300, json: async () => h.probeUrl.body };throw new Error(`unexpected fetch ${url}`);},-- openDb: h.openDb || ((p) => new Database(p, { readonly: true, fileMustExist: true })),++ openDb: h.openDb || ((p) => (p === 'pglite:openre' ? keepOpen(dbs.openre) : new Database(p, { readonly: true, fileMustExist: true }))),};deps.h = h;return deps;@@@ -144,8 +149,8 @@ function assertNoSecret(report) {// ── tests ──────────────────────────────────────────────────────────────────t('a host ready for the first slot: every check passes; on the host the port check is MANUAL', async () => {-- const dbs = makeDbs();-- addSubscription(dbs);++ const dbs = await makeDbs();++ await addSubscription(dbs);const deps = host(dbs);const report = await preflight(opts(dbs), deps);const r = byId(report);@@@ -164,7 +169,7 @@ t('a host ready for the first slot: every check passes; on the host the port che});t('the host as it is on 2026-09-23: old release, loopback bind, Live env and subscription missing', async () => {-- const dbs = makeDbs();++ const dbs = await makeDbs();const deps = host(dbs, {release: '655b98a10aaa',originMain: '450f87c3b2a1',@@@ -192,7 +197,7 @@ t('the host as it is on 2026-09-23: old release, loopback bind, Live env and sub});t('release: --expect-release wins over origin/main; --require-commit none skips the ancestry check', async () => {-- const dbs = makeDbs();++ const dbs = await makeDbs();const deps = host(dbs, { originMain: 'bbbbbbbbbbbb', containsCommit: false });let r = byId(await preflight(opts(dbs, ['--only', 'release', '--expect-release', RELEASE, '--require-commit', 'none']), deps));assert.strictEqual(r.release.status, 'PASS');@@@ -204,7 +209,7 @@ t('release: --expect-release wins over origin/main; --require-commit none skips});t('bind: public in the env but the running generation still listens on loopback → deploy.sh workers', async () => {-- const dbs = makeDbs();++ const dbs = await makeDbs();const r = byId(await preflight(opts(dbs, ['--only', 'bind']), host(dbs, { ss: 'LISTEN 0 511 127.0.0.1:1936 0.0.0.0:*\n' })));assert.strictEqual(r.bind.status, 'FAIL');assert.match(r.bind.detail, /deploy\.sh workers/);@@@ -213,7 +218,7 @@ t('bind: public in the env but the running generation still listens on loopback});t('env: OPENRE_DRILL in the production env file fails; so do a wrong port or host', async () => {-- const dbs = makeDbs();++ const dbs = await makeDbs();const base = host(dbs).h.openreEnv;let r = byId(await preflight(opts(dbs, ['--only', 'env']), host(dbs, { openreEnv: { ...base, OPENRE_DRILL: '1' } })));assert.strictEqual(r.env.status, 'FAIL');@@@ -227,7 +232,7 @@ t('env: OPENRE_DRILL in the production env file fails; so do a wrong port or hos});t('dns: a proxied (foreign) address fails; AAAA with an IPv4-only bind warns; off the host --host-ip decides', async () => {-- const dbs = makeDbs();++ const dbs = await makeDbs();let r = byId(await preflight(opts(dbs, ['--only', 'dns']), host(dbs, { a4: ['104.21.3.4'] })));assert.strictEqual(r.dns.status, 'FAIL');assert.match(r.dns.detail, /Cloudflare-proxied/);@@@ -245,7 +250,7 @@ t('dns: a proxied (foreign) address fails; AAAA with an IPv4-only bind warns; of});t('port from outside the host: handshake = PASS, refused = no public listener, timeout = provider edge', async () => {-- const dbs = makeDbs();++ const dbs = await makeDbs();const laptop = { wlan0: [{ address: '10.0.0.9', internal: false }] };const run = (probe) => preflight(opts(dbs, ['--only', 'port']), host(dbs, { interfaces: laptop, probe })).then(byId);let r = await run({ ok: true, rtmp: true, ms: 40 });@@@ -262,7 +267,7 @@ t('port from outside the host: handshake = PASS, refused = no public listener, t});t('port with --probe-url: open/closed from the probe, anything else MANUAL', async () => {-- const dbs = makeDbs();++ const dbs = await makeDbs();const url = 'https://probe.example/check?host={host}&port={port}';const run = (probeUrl) => {const deps = host(dbs, { probeUrl });@@@ -281,7 +286,7 @@ t('port with --probe-url: open/closed from the probe, anything else MANUAL', asy});t('service: not ready, drill mode, a worker on an old release, no lease, relay off, draining', async () => {-- const dbs = makeDbs();++ const dbs = await makeDbs();const ready = host(dbs).h.ready;const run = (body, status = 200, argv = []) => preflight(opts(dbs, ['--only', 'release,service', ...argv]), host(dbs, { ready: { status, body } })).then(byId);let r = await run({ ...ready.body, status: 'not_ready', checks: { db: true, key: false } }, 503);@@@ -298,9 +303,7 @@ t('service: not ready, drill mode, a worker on an old release, no lease, relay or = await run({ ...ready.body, workers: [{ kind: 'restream', generation: 2, state: 'ready' }] });assert.match(r.service.detail, /0 ready rtmp-ingest workers/);// the ready generation runs another release than `current`-- const db = new Database(dbs.openrePath);-- db.prepare("UPDATE workers SET release = '655b98a10aaa' WHERE kind = 'rtmp-ingest'").run();-- db.close();++ await dbs.openre.prepare("UPDATE workers SET release = '655b98a10aaa' WHERE kind = 'rtmp-ingest'").run();r = await run(ready.body);assert.strictEqual(r.service.status, 'FAIL');assert.match(r.service.detail, /rtmp-ingest#2 runs release 655b98a10aaa, current is aaaaaaaaaaaa: deploy\.sh workers/);@@@ -309,7 +312,7 @@ t('service: not ready, drill mode, a worker on an old release, no lease, relay o});t('live-env: Live running without the new settings must be restarted; a short secret is refused', async () => {-- const dbs = makeDbs();++ const dbs = await makeDbs();const env = host(dbs).h.liveEnv;let r = byId(await preflight(opts(dbs, ['--only', 'live-env']), host(dbs, { runningLiveEnv: { OV_OAUTH_CLIENT_SECRET: CLIENT_SECRET } })));assert.strictEqual(r['live-env'].status, 'FAIL');@@@ -324,29 +327,29 @@ t('live-env: Live running without the new settings must be restarted; a short se});t('events: wrong secret, disabled, duplicates, dead deliveries fail; waiting deliveries warn; secrets never printed', async () => {-- let dbs = makeDbs();-- addSubscription(dbs, { secret: 'x'.repeat(64) });++ let dbs = await makeDbs();++ await addSubscription(dbs, { secret: 'x'.repeat(64) });let r = byId(await preflight(opts(dbs, ['--only', 'events']), host(dbs)));assert.strictEqual(r.events.status, 'FAIL');assert.match(r.events.detail, /secret is not Live's OPENRE_EVENTS_SECRET/);assertNoSecret({ results: Object.values(r) });-- dbs = makeDbs();-- addSubscription(dbs, { enabled: 0 });-- addDelivery(dbs, 'dead', 2);++ dbs = await makeDbs();++ await addSubscription(dbs, { enabled: 0 });++ await addDelivery(dbs, 'dead', 2);r = byId(await preflight(opts(dbs, ['--only', 'events']), host(dbs)));assert.match(r.events.detail, /disabled/);assert.match(r.events.detail, /2 dead deliveries/);-- dbs = makeDbs();-- addSubscription(dbs);-- addSubscription(dbs, { id: 'sub_2' });++ dbs = await makeDbs();++ await addSubscription(dbs);++ await addSubscription(dbs, { id: 'sub_2' });r = byId(await preflight(opts(dbs, ['--only', 'events']), host(dbs)));assert.match(r.events.detail, /2 identical subscriptions/);-- dbs = makeDbs();-- addSubscription(dbs);-- addDelivery(dbs, 'failed', 3);++ dbs = await makeDbs();++ await addSubscription(dbs);++ await addDelivery(dbs, 'failed', 3);r = byId(await preflight(opts(dbs, ['--only', 'events']), host(dbs)));assert.strictEqual(r.events.status, 'WARN');assert.match(r.events.detail, /3 deliveries waiting or retrying/);@@@ -356,7 +359,7 @@ t('events: wrong secret, disabled, duplicates, dead deliveries fail; waiting del});t('db: integrity_check and foreign keys from a read-only handle', async () => {-- const dbs = makeDbs();++ const dbs = await makeDbs();let r = byId(await preflight(opts(dbs, ['--only', 'db']), host(dbs)));assert.strictEqual(r.db.status, 'PASS');assert.match(r.db.detail, /stream_definitions 1, ingest_keys 0, ingest_sessions 0, destinations 0, migration_map 1/);@@@ -367,20 +370,20 @@ t('db: integrity_check and foreign keys from a read-only handle', async () => {r = byId(await preflight(opts(dbs, ['--only', 'db']), host(dbs, { openDb: corrupt })));assert.strictEqual(r.db.status, 'FAIL');assert.match(r.db.detail, /Page 7 is never used/);-- // the real handle really is read-only-- const ro = new Database(dbs.openrePath, { readonly: true, fileMustExist: true });-- assert.throws(() => ro.prepare("DELETE FROM migration_map").run(), /readonly/);-- ro.close();++ // the preflight's PostgreSQL transaction really is read-only++ const w = await withDb(host(dbs), dbs.openrePath, async (db) => await db.prepare('DELETE FROM migration_map').run());++ assert.match(w.error, /read-only transaction/);++ assert.strictEqual(Number((await dbs.openre.prepare('SELECT count(*) AS n FROM migration_map').get()).n), 1);});t('slot: offline RTMP slot with a subject and a migrated definition passes; live now / no subject fail', async () => {-- const dbs = makeDbs();++ const dbs = await makeDbs();let r = byId(await preflight(opts(dbs, ['--only', 'slot', '--slot', '12']), host(dbs)));assert.strictEqual(r.slot.status, 'PASS', r.slot.detail);assert.match(r.slot.detail, /slot 12 \(@rehearsal, rtmp\/obs\) authority live; OpenRe str_1 \(active, mirrored\); 0 destination\(s\)/);const l = new Database(dbs.livePath);-- l.prepare('INSERT INTO streams (managed_stream_id, is_live) VALUES (12, 1)').run();-- l.prepare('DELETE FROM linked_accounts').run();++ await l.prepare('INSERT INTO streams (managed_stream_id, is_live) VALUES (12, 1)').run();++ await l.prepare('DELETE FROM linked_accounts').run();l.close();r = byId(await preflight(opts(dbs, ['--only', 'slot', '--slot', '12']), host(dbs)));assert.strictEqual(r.slot.status, 'FAIL');@@@ -25,14 +25,14 @@ async function setup({ media: mediaHandler, recording_mode = 'vod' }) {const calls = [];const media = await stub((req, res, body) => { calls.push({ method: req.method, url: req.url, body, auth: req.headers.authorization }); mediaHandler(req, res, body, calls); });const clock = manualClock();-- const rt = runtime({ clock, env: { MEDIA_URL: media.url, MEDIA_API_KEY: 'k-live', OPENRE_RECORDING_START_DELAY_MS: '0' } });++ const rt = await runtime({ clock, env: { MEDIA_URL: media.url, MEDIA_API_KEY: 'k-live', OPENRE_RECORDING_START_DELAY_MS: '0' } });const client = createMediaClient({ config: rt.config });const coordinator = createCoordinator({ rt, media: client, log: silent });-- const { definition, key } = rt.store.definitions.create({ owner_subject: OWNER, title: 'Rec', recording_mode, external_refs: [{ service: 'live', type: 'user', id: '8' }] });-- const w = rt.store.workers.register({ kind: 'rtmp-ingest', endpoints: { rtmpPlayPort: 19390, flvPort: 19391 } });-- rt.store.workers.ready(w.id);-- const a = rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker: rt.store.workers.get(w.id) });-- rt.store.sessions.transition(a.session.id, 'live');++ const { definition, key } = await rt.store.definitions.create({ owner_subject: OWNER, title: 'Rec', recording_mode, external_refs: [{ service: 'live', type: 'user', id: '8' }] });++ const w = await rt.store.workers.register({ kind: 'rtmp-ingest', endpoints: { rtmpPlayPort: 19390, flvPort: 19391 } });++ await rt.store.workers.ready(w.id);++ const a = await rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker: await rt.store.workers.get(w.id) });++ await rt.store.sessions.transition(a.session.id, 'live');return { rt, clock, coordinator, session: a.session, calls, media };}@@@ -52,12 +52,12 @@ t('a live session gets one recording request; finalize when it ends (Media finalassert.strictEqual(calls[0].body.user_id, 8);assert.strictEqual(calls[0].body.visibility, 'public');assert.strictEqual(calls[1].body.rtmp_url, `rtmp://127.0.0.1:19390/live/${session.id}`);-- assert.strictEqual(rt.store.recordings.bySession(session.id).state, 'recording');-- assert.deepStrictEqual(outboxTypes(rt.db).filter(x => x.startsWith('openre.recording')), ['openre.recording.requested']);-- rt.store.sessions.finish(session.id, { reason: 'publisher_disconnected' });++ assert.strictEqual((await rt.store.recordings.bySession(session.id)).state, 'recording');++ assert.deepStrictEqual((await outboxTypes(rt.db)).filter(x => x.startsWith('openre.recording')), ['openre.recording.requested']);++ await rt.store.sessions.finish(session.id, { reason: 'publisher_disconnected' });await coordinator.tick();assert.strictEqual(calls.pop().url, '/api/v1/live/vods/501/finalize');-- assert.strictEqual(rt.store.recordings.bySession(session.id).state, 'finalized');++ assert.strictEqual((await rt.store.recordings.bySession(session.id)).state, 'finalized');await media.close();});@@@ -73,15 +73,15 @@ t('Media refusing for disk space: the empty VOD shell is deleted and the request});await coordinator.tick();assert.deepStrictEqual(calls.map(c => c.method), ['POST', 'POST', 'DELETE'], 'create, ingest refused, shell deleted');-- const rec = rt.store.recordings.bySession(session.id);++ const rec = await rt.store.recordings.bySession(session.id);assert.strictEqual(rec.state, 'pending');assert.match(rec.last_error, /Disk/);assert.ok(rec.next_attempt_at >= clock.now() + 5 * 60 * 1000, 'disk refusals wait five minutes');refuse = false;clock.advance(5 * 60 * 1000 + 1);-- rt.store.workers.heartbeat(session.worker_id); // the ingest worker is alive throughout++ await rt.store.workers.heartbeat(session.worker_id); // the ingest worker is alive throughoutawait coordinator.tick();-- assert.strictEqual(rt.store.recordings.bySession(session.id).state, 'recording');++ assert.strictEqual((await rt.store.recordings.bySession(session.id)).state, 'recording');await media.close();});@@@ -96,16 +96,16 @@ t('clips-only recordings are deleted after finalising; a session that ends first});await coordinator.tick();assert.strictEqual(calls[0].body.clips_only, true);-- rt.store.sessions.finish(session.id, { reason: 'x' });++ await rt.store.sessions.finish(session.id, { reason: 'x' });await coordinator.tick();assert.deepStrictEqual(calls.slice(-2).map(c => `${c.method} ${c.url}`), ['POST /api/v1/live/vods/700/finalize', 'DELETE /api/v1/live/vods/700']);await media.close();const s2 = await setup({ media: (req, res) => reply(res, 500) });-- s2.rt.store.sessions.finish(s2.session.id, { reason: 'gone' });-- s2.rt.db.prepare("INSERT INTO recordings (id, session_id, mode, state, created_at, updated_at) VALUES ('rec_01J0000000000000000000000A', ?, 'vod', 'pending', 0, 0)").run(s2.session.id);++ await s2.rt.store.sessions.finish(s2.session.id, { reason: 'gone' });++ await s2.rt.db.prepare("INSERT INTO recordings (id, session_id, mode, state, created_at, updated_at) VALUES ('rec_01J0000000000000000000000A', ?, 'vod', 'pending', 0, 0)").run(s2.session.id);await s2.coordinator.tick();-- assert.strictEqual(s2.rt.store.recordings.bySession(s2.session.id).state, 'cancelled');++ assert.strictEqual((await s2.rt.store.recordings.bySession(s2.session.id)).state, 'cancelled');assert.strictEqual(s2.calls.length, 0);await s2.media.close();});@@@ -127,29 +127,29 @@ t('the relay publishes outbox rows with a Network service token; Events down meaassert.strictEqual(p.get('audience'), 'openvibe.events');reply(res, 200, { access_token: 'svc-token', token_type: 'Bearer', expires_in: 300 });});-- const rt = runtime({ env: { EVENTS_URL: events.url, OV_NETWORK_INTERNAL_URL: network.url, OV_OAUTH_CLIENT_SECRET: 's3cret' } });++ const rt = await runtime({ env: { EVENTS_URL: events.url, OV_NETWORK_INTERNAL_URL: network.url, OV_OAUTH_CLIENT_SECRET: 's3cret' } });assert.strictEqual(rt.events.configured, true);-- const { definition } = rt.store.definitions.create({ owner_subject: OWNER });-- rt.store.definitions.rotateKey(definition.id, { rotated_by: OWNER });++ const { definition } = await rt.store.definitions.create({ owner_subject: OWNER });++ await rt.store.definitions.rotateKey(definition.id, { rotated_by: OWNER });await rt.events.outbox.flush();-- assert.strictEqual(rt.events.outbox.pending(), 1, 'kept while Events is down');++ assert.strictEqual(await rt.events.outbox.pending(), 1, 'kept while Events is down');down = false;-- rt.db.prepare('UPDATE event_outbox SET next_attempt_at = 0').run();++ await rt.db.prepare('UPDATE event_outbox SET next_attempt_at = 0').run();await rt.events.outbox.flush();-- assert.strictEqual(rt.events.outbox.pending(), 0);++ assert.strictEqual(await rt.events.outbox.pending(), 0);assert.strictEqual(got[0].event_type, 'openre.key.rotated');assert.strictEqual(got[0].source, 'openre');await events.close();await network.close();});--t('without EVENTS_URL the relay is off but every row is still written with its change', () => {-- const rt = runtime();++t('without EVENTS_URL the relay is off but every row is still written with its change', async () => {++ const rt = await runtime();assert.strictEqual(rt.events.configured, false);assert.strictEqual(rt.events.start(), false);-- const { definition } = rt.store.definitions.create({ owner_subject: OWNER });-- rt.store.definitions.rotateKey(definition.id);-- assert.strictEqual(rt.events.status().pending, 1);++ const { definition } = await rt.store.definitions.create({ owner_subject: OWNER });++ await rt.store.definitions.rotateKey(definition.id);++ assert.strictEqual((await rt.events.status()).pending, 1);});t.run();@@@ -13,7 +13,7 @@ const assert = require('assert');const http = require('http');const { spawn } = require('child_process');const {-- tmpDir, testEnv, child, waitFor, request, userToken, freePort, sleep, suite, hasFfmpeg, publish, rtmpSink, OWNER,++ tmpDir, testEnv, multiProcess, child, waitFor, request, userToken, freePort, sleep, suite, hasFfmpeg, publish, rtmpSink, OWNER,} = require('./helpers');if (!hasFfmpeg()) {@@@ -21,6 +21,10 @@ if (!hasFfmpeg()) {process.exit(0);}++if (!multiProcess()) {++ console.log('rtmp-e2e: skipped (the four OpenRe processes share one database: PostgreSQL only, npm run test:pg)');++ process.exit(0);++}const t = suite('rtmp-e2e');const token = userToken({ subjectId: OWNER });const procs = [];@@@ -104,7 +108,7 @@ t('setup: stubs and the four OpenRe processes', async () => {});rtmpPort = await freePort();const dir = tmpDir();-- env = testEnv(dir, {++ env = await testEnv(dir, {PORT: String(await freePort()),OPENRE_RTMP_PORT: String(rtmpPort),OPENRE_RTMP_INTERNAL_PORT_MIN: String(20000 + Math.floor(Math.random() * 20000)),@@@ -153,7 +157,7 @@ t('a wrong key is refused; the right key goes live on generation 1', async () =>assert.notStrictEqual(w.code, 0, 'ffmpeg with a wrong key fails');pubA = publish(`rtmp://127.0.0.1:${rtmpPort}/live/${keyA}`);procs.push(pubA);-- sessionA = await waitFor(() => session(streamA), { what: 'session A live', timeoutMs: 15000 });++ sessionA = await waitFor(async () => await session(streamA), { what: 'session A live', timeoutMs: 15000 });assert.strictEqual(sessionA.worker.generation, 1);});@@@ -171,7 +175,7 @@ t('the working destination goes live, the dead one fails, the session stays liveconst o = r.body.outputs || [];return o.length === 2 && o.some(x => x.state === 'live') && o.some(x => x.state === 'failed') ? o : null;}, { what: 'outputs live + failed', timeoutMs: 40000 });-- const failed = outputs.find(o => o.state === 'failed');++ const failed = await outputs.find(o => o.state === 'failed');assert.match(failed.last_error, /connect|closed|refused|rapid/i);assert.strictEqual((await session(streamA)).id, sessionA.id, 'the source session is still live');assert.strictEqual(pubA.exitCode, null, 'the encoder is still connected');@@@ -201,7 +205,7 @@ t('restarting the API during the broadcast ends nothing; playback works throughassert.strictEqual(s && s.id, sessionA.id);assert.strictEqual(pubA.exitCode, null);const outputs = (await request(base, 'GET', `/api/v1/sessions/${sessionA.id}/outputs`, { token })).body.outputs;-- assert.ok(outputs.some(o => o.state === 'live'), 'the restream kept running');++ assert.ok(await outputs.some(o => o.state === 'live'), 'the restream kept running');const head = await new Promise((resolve, reject) => {http.get(`${base}/play/${sessionA.id}.flv`, (res) => {assert.strictEqual(res.statusCode, 200);@@@ -224,7 +228,7 @@ t('a new rtmp-ingest generation takes new sessions; generation 1 drains but keepstreamB = r.body.stream.id;pubB = publish(`rtmp://127.0.0.1:${rtmpPort}/live/${r.body.key.key}`);procs.push(pubB);-- const sB = await waitFor(() => session(streamB), { what: 'session B live', timeoutMs: 15000 });++ const sB = await waitFor(async () => await session(streamB), { what: 'session B live', timeoutMs: 15000 });assert.strictEqual(sB.worker.generation, 2, 'new sessions reach the newest generation');const sA = await session(streamA);assert.strictEqual(sA.id, sessionA.id);@@@ -8,12 +8,12 @@ const { NEXT, canTransition } = require('../server/state-machine');const t = suite('sessions');--function setup(clock = manualClock()) {-- const rt = runtime({ clock });-- const { definition, key } = rt.store.definitions.create({ owner_subject: OWNER, title: 'Show', external_refs: [{ service: 'live', type: 'managed_stream', id: '12' }] });-- const w = rt.store.workers.register({ kind: 'rtmp-ingest', endpoints: { publicPort: 1936, rtmpPlayPort: 19360, flvPort: 19361 } });-- rt.store.workers.ready(w.id);-- return { rt, clock, definition, key, worker: rt.store.workers.get(w.id) };++async function setup(clock = manualClock()) {++ const rt = await runtime({ clock });++ const { definition, key } = await rt.store.definitions.create({ owner_subject: OWNER, title: 'Show', external_refs: [{ service: 'live', type: 'managed_stream', id: '12' }] });++ const w = await rt.store.workers.register({ kind: 'rtmp-ingest', endpoints: { publicPort: 1936, rtmpPlayPort: 19360, flvPort: 19361 } });++ await rt.store.workers.ready(w.id);++ return { rt, clock, definition, key, worker: await rt.store.workers.get(w.id) };}t('the transition table is exactly starting→live→ending→ended with failed from any open state', () => {@@@ -23,18 +23,18 @@ t('the transition table is exactly starting→live→ending→ended with failedassert.ok(!canTransition('failed', 'ended'));});--t('a clean session emits started then ended; envelopes are valid events.event-envelope@1', () => {-- const { rt, clock, definition, key, worker } = setup();-- const a = rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker });++t('a clean session emits started then ended; envelopes are valid events.event-envelope@1', async () => {++ const { rt, clock, definition, key, worker } = await setup();++ const a = await rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker });assert.strictEqual(a.session.state, 'starting');-- assert.deepStrictEqual(outboxTypes(rt.db), [], 'starting is internal');-- assert.ok(rt.store.sessions.transition(a.session.id, 'live', { reason: 'media_flowing' }).ok);++ assert.deepStrictEqual(await outboxTypes(rt.db), [], 'starting is internal');++ assert.ok((await rt.store.sessions.transition(a.session.id, 'live', { reason: 'media_flowing' })).ok);clock.advance(90 * 1000);-- const f = rt.store.sessions.finish(a.session.id, { reason: 'publisher_disconnected' });++ const f = await rt.store.sessions.finish(a.session.id, { reason: 'publisher_disconnected' });assert.ok(f.ok);assert.strictEqual(f.session.state, 'ended');-- assert.deepStrictEqual(outboxTypes(rt.db), ['openre.session.started', 'openre.session.ended']);-- const [started, ended] = outboxEnvelopes(rt.db);++ assert.deepStrictEqual(await outboxTypes(rt.db), ['openre.session.started', 'openre.session.ended']);++ const [started, ended] = await outboxEnvelopes(rt.db);for (const ev of [started, ended]) {const v = contracts.validate('events.event-envelope@1', ev);assert.ok(v.valid, JSON.stringify(v.errors));@@@ -47,78 +47,78 @@ t('a clean session emits started then ended; envelopes are valid events.event-enassert.deepStrictEqual(started.payload.external_refs, [{ service: 'live', type: 'managed_stream', id: '12', label: null }]);assert.strictEqual(started.payload.playback.flv.internal_url, `http://127.0.0.1:19361/live/${a.session.id}.flv`);assert.strictEqual(ended.payload.duration_seconds, 90);-- assert.deepStrictEqual(rt.store.sessions.transitions(a.session.id).map(x => x.to_state), ['starting', 'live', 'ending', 'ended']);++ assert.deepStrictEqual((await rt.store.sessions.transitions(a.session.id)).map(x => x.to_state), ['starting', 'live', 'ending', 'ended']);});--t('a publisher that leaves during the handshake ends without events; failures always emit', () => {-- const { rt, definition, key, worker } = setup();-- const a = rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker });-- rt.store.sessions.finish(a.session.id, { reason: 'publisher_disconnected' });-- assert.deepStrictEqual(outboxTypes(rt.db), []);-- const b = rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker });-- rt.store.sessions.transition(b.session.id, 'failed', { reason: 'worker_lost' });-- const ev = outboxEnvelopes(rt.db).pop();++t('a publisher that leaves during the handshake ends without events; failures always emit', async () => {++ const { rt, definition, key, worker } = await setup();++ const a = await rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker });++ await rt.store.sessions.finish(a.session.id, { reason: 'publisher_disconnected' });++ assert.deepStrictEqual(await outboxTypes(rt.db), []);++ const b = await rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker });++ await rt.store.sessions.transition(b.session.id, 'failed', { reason: 'worker_lost' });++ const ev = (await outboxEnvelopes(rt.db)).pop();assert.strictEqual(ev.event_type, 'openre.session.failed');assert.strictEqual(ev.payload.was_live, false);assert.strictEqual(ev.payload.failure_reason, 'worker_lost');});--t('invalid transitions change nothing (a worker and the coordinator racing)', () => {-- const { rt, definition, key, worker } = setup();-- const a = rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker });-- rt.store.sessions.transition(a.session.id, 'live');-- rt.store.sessions.transition(a.session.id, 'failed', { reason: 'lease_expired' });-- const before = outboxTypes(rt.db).length;-- const r = rt.store.sessions.finish(a.session.id, { reason: 'publisher_disconnected' });++t('invalid transitions change nothing (a worker and the coordinator racing)', async () => {++ const { rt, definition, key, worker } = await setup();++ const a = await rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker });++ await rt.store.sessions.transition(a.session.id, 'live');++ await rt.store.sessions.transition(a.session.id, 'failed', { reason: 'lease_expired' });++ const before = (await outboxTypes(rt.db)).length;++ const r = await rt.store.sessions.finish(a.session.id, { reason: 'publisher_disconnected' });assert.strictEqual(r.ok, false);assert.strictEqual(r.code, 'invalid_transition');-- assert.strictEqual(rt.store.sessions.get(a.session.id).state, 'failed');-- assert.strictEqual(outboxTypes(rt.db).length, before);++ assert.strictEqual((await rt.store.sessions.get(a.session.id)).state, 'failed');++ assert.strictEqual((await outboxTypes(rt.db)).length, before);});--t('one open session per stream: a second publisher is refused while the first holds its lease', () => {-- const { rt, clock, definition, key, worker } = setup();-- const a = rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker });-- assert.strictEqual(rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker }).error, 'duplicate_publisher');++t('one open session per stream: a second publisher is refused while the first holds its lease', async () => {++ const { rt, clock, definition, key, worker } = await setup();++ const a = await rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker });++ assert.strictEqual((await rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker })).error, 'duplicate_publisher');// Lease expired (worker died): the stale session fails and the new publish is admitted.clock.advance(rt.config.workers.leaseMs + 1);-- const b = rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker });++ const b = await rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker });assert.ok(b.session);-- assert.strictEqual(rt.store.sessions.get(a.session.id).state, 'failed');-- assert.strictEqual(rt.store.sessions.get(a.session.id).failure_reason, 'lease_expired');++ assert.strictEqual((await rt.store.sessions.get(a.session.id)).state, 'failed');++ assert.strictEqual((await rt.store.sessions.get(a.session.id)).failure_reason, 'lease_expired');});--t('heartbeats renew the lease of every session the worker owns', () => {-- const { rt, clock, definition, key, worker } = setup();-- const a = rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker });-- const lease1 = rt.store.sessions.get(a.session.id).lease_expires_at;++t('heartbeats renew the lease of every session the worker owns', async () => {++ const { rt, clock, definition, key, worker } = await setup();++ const a = await rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker });++ const lease1 = (await rt.store.sessions.get(a.session.id)).lease_expires_at;clock.advance(5000);-- rt.store.workers.heartbeat(worker.id);-- assert.strictEqual(rt.store.sessions.get(a.session.id).lease_expires_at, lease1 + 5000);++ await rt.store.workers.heartbeat(worker.id);++ assert.strictEqual((await rt.store.sessions.get(a.session.id)).lease_expires_at, lease1 + 5000);});--t('ending a session asks its outputs to stop; an output never moves the session', () => {-- const { rt, definition, key, worker } = setup();-- const dest = rt.store.outputs.createDestination(definition.id, { platform: 'custom', server_url: 'rtmp://127.0.0.1:1/app', stream_key: 'k1234' });-- rt.store.outputs.createDestination(definition.id, { platform: 'custom', server_url: 'rtmp://127.0.0.1:2/app', stream_key: 'k5678' });-- const a = rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker });-- rt.store.sessions.transition(a.session.id, 'live');-- assert.strictEqual(rt.store.outputs.ensureAutoOutputs(), 2);-- const [out, other] = rt.store.outputs.outputsOfSession(a.session.id);-- rt.store.outputs.report(other.id, { state: 'live' });-- rt.store.outputs.report(out.id, { state: 'failed', last_error: 'boom' });-- assert.strictEqual(rt.store.sessions.get(a.session.id).state, 'live', 'a failed output leaves the session live');-- assert.ok(outboxTypes(rt.db).includes('openre.output.failed'));-- rt.store.sessions.finish(a.session.id, { reason: 'publisher_disconnected' });-- assert.strictEqual(rt.db.prepare('SELECT desired FROM outputs WHERE id = ?').get(other.id).desired, 'stop', 'the running output is told to stop');++t('ending a session asks its outputs to stop; an output never moves the session', async () => {++ const { rt, definition, key, worker } = await setup();++ const dest = await rt.store.outputs.createDestination(definition.id, { platform: 'custom', server_url: 'rtmp://127.0.0.1:1/app', stream_key: 'k1234' });++ await rt.store.outputs.createDestination(definition.id, { platform: 'custom', server_url: 'rtmp://127.0.0.1:2/app', stream_key: 'k5678' });++ const a = await rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker });++ await rt.store.sessions.transition(a.session.id, 'live');++ assert.strictEqual(await rt.store.outputs.ensureAutoOutputs(), 2);++ const [out, other] = await rt.store.outputs.outputsOfSession(a.session.id);++ await rt.store.outputs.report(other.id, { state: 'live' });++ await rt.store.outputs.report(out.id, { state: 'failed', last_error: 'boom' });++ assert.strictEqual((await rt.store.sessions.get(a.session.id)).state, 'live', 'a failed output leaves the session live');++ assert.ok((await outboxTypes(rt.db)).includes('openre.output.failed'));++ await rt.store.sessions.finish(a.session.id, { reason: 'publisher_disconnected' });++ assert.strictEqual((await rt.db.prepare('SELECT desired FROM outputs WHERE id = ?').get(other.id)).desired, 'stop', 'the running output is told to stop');assert.ok(dest.id);});--t('an end request is recorded for the owning worker to act on', () => {-- const { rt, definition, key, worker } = setup();-- const a = rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker });-- assert.ok(rt.store.sessions.requestEnd(a.session.id, OWNER));-- assert.deepStrictEqual(rt.store.sessions.endRequestsFor(worker.id).map(s => s.id), [a.session.id]);++t('an end request is recorded for the owning worker to act on', async () => {++ const { rt, definition, key, worker } = await setup();++ const a = await rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker });++ assert.ok(await rt.store.sessions.requestEnd(a.session.id, OWNER));++ assert.deepStrictEqual((await rt.store.sessions.endRequestsFor(worker.id)).map(s => s.id), [a.session.id]);});t.run();@@@ -40,7 +40,7 @@ t('setup: a definition made for Live slot 12, as Live does at the switch', asyncassert.strictEqual(created.status, 201);streamId = created.body.stream.id;key = created.body.key.key;-- assert.ok(api.rt.store.definitions.resolveIngestKey(key, 'rtmp').definition);++ assert.ok((await api.rt.store.definitions.resolveIngestKey(key, 'rtmp')).definition);});t('--state disabled: the definition is disabled and its key refused at the handshake', async () => {@@@ -48,7 +48,7 @@ t('--state disabled: the definition is disabled and its key refused at the handsconst r = await run({ slot: '12', state: 'disabled', env, fetchImpl, log });assert.deepStrictEqual(r, { id: streamId, state: 'disabled', changed: true, open_session: null });assert.deepStrictEqual(tokenRequests.at(-1), { grant_type: 'client_credentials', client_id: 'live', client_secret: CLIENT_SECRET, audience: 'openvibe.openre' });-- assert.deepStrictEqual(api.rt.store.definitions.resolveIngestKey(key, 'rtmp'), { error: 'stream_disabled' });++ assert.deepStrictEqual(await api.rt.store.definitions.resolveIngestKey(key, 'rtmp'), { error: 'stream_disabled' });assert.match(log.lines.join('\n'), new RegExp(`${streamId} \\(live:managed_stream:12\\): active → disabled`));for (const l of log.lines) { assert.ok(!l.includes(key)); assert.ok(!l.includes(CLIENT_SECRET)); }const again = await run({ slot: '12', state: 'disabled', env, fetchImpl, log });@@@ -58,7 +58,7 @@ t('--state disabled: the definition is disabled and its key refused at the handst('--state active: the same key works again (re-switching the slot later)', async () => {const r = await run({ slot: '12', state: 'active', env, fetchImpl, log: () => {} });assert.strictEqual(r.state, 'active');-- assert.ok(api.rt.store.definitions.resolveIngestKey(key, 'rtmp').definition);++ assert.ok((await api.rt.store.definitions.resolveIngestKey(key, 'rtmp')).definition);});t('a slot without a definition is an error; arguments are checked', async () => {@@@ -9,26 +9,28 @@const path = require('path');if (require.main === module) {-- require('dotenv').config({ path: process.env.OPENRE_ENV_FILE || path.join(process.cwd(), '.env') });-- const { load, exitIfDrill } = require('../server/config');-- const { openRuntime } = require('../server/store');-- const { createCoordinator } = require('../server/coordinator');-- const { createMediaClient } = require('../server/media-client');-- const config = load();-- exitIfDrill(config, 'openre-session-coordinator');-- const rt = openRuntime({ config });-- const media = createMediaClient({ config });-- const lineage = require('../server/lineage').createLineage({ db: rt.db, config });-- const coordinator = createCoordinator({ rt, media, lineage });-- coordinator.start();-- console.log(`[coordinator] running (events relay ${rt.events.configured ? 'on' : 'off: EVENTS_URL/OV_OAUTH_CLIENT_SECRET not set'}, recordings ${media.configured ? `on → ${config.media.url} app ${config.media.appId}` : 'off'}, Live lineage ${lineage.enabled ? 'on' : 'off'})`);-- const prune = setInterval(() => { try { rt.events.outbox.prune(); } catch { /* next time */ } }, 6 * 60 * 60 * 1000);-- prune.unref();-- const shutdown = (sig) => {-- console.log(`[coordinator] ${sig}: stopping`);-- coordinator.stop().then(() => { rt.db.close(); process.exit(0); }, () => process.exit(1));-- setTimeout(() => process.exit(1), 10000).unref();-- };-- process.on('SIGTERM', () => shutdown('SIGTERM'));-- process.on('SIGINT', () => shutdown('SIGINT'));++ (async () => {++ require('dotenv').config({ path: process.env.OPENRE_ENV_FILE || path.join(process.cwd(), '.env') });++ const { load, exitIfDrill } = require('../server/config');++ const { openRuntime } = require('../server/store');++ const { createCoordinator } = require('../server/coordinator');++ const { createMediaClient } = require('../server/media-client');++ const config = load();++ exitIfDrill(config, 'openre-session-coordinator');++ const rt = await openRuntime({ config });++ const media = createMediaClient({ config });++ const lineage = require('../server/lineage').createLineage({ db: rt.db, config });++ const coordinator = createCoordinator({ rt, media, lineage });++ coordinator.start();++ console.log(`[coordinator] running (events relay ${rt.events.configured ? 'on' : 'off: EVENTS_URL/OV_OAUTH_CLIENT_SECRET not set'}, recordings ${media.configured ? `on → ${config.media.url} app ${config.media.appId}` : 'off'}, Live lineage ${lineage.enabled ? 'on' : 'off'})`);++ const prune = setInterval(() => { try { rt.events.outbox.prune(); } catch { /* next time */ } }, 6 * 60 * 60 * 1000);++ prune.unref();++ const shutdown = (sig) => {++ console.log(`[coordinator] ${sig}: stopping`);++ coordinator.stop().then(() => { rt.db.close(); process.exit(0); }, () => process.exit(1));++ setTimeout(() => process.exit(1), 10000).unref();++ };++ process.on('SIGTERM', () => shutdown('SIGTERM'));++ process.on('SIGINT', () => shutdown('SIGINT'));++ })().catch((err) => { console.error(`[coordinator] failed to start: ${err && err.stack || err}`); process.exit(1); });}@@@ -1,3 +1,3 @@'use strict';/** openre-jsmpeg: worker interface only, the transport is not ported yet (see unported-transport.js). */--if (require.main === module) require('./unported-transport').main('jsmpeg');++if (require.main === module) require('./unported-transport').main('jsmpeg').catch((err) => { console.error(`[jsmpeg] failed to start: ${err && err.stack || err}`); process.exit(1); });@@@ -24,39 +24,39 @@ function createRestreamWorker({ rt, log = console, exit = (code) => process.exitconst runtime = createWorkerRuntime({rt, kind: 'restream', log, exit,hooks: {-- onDrainDeadline: () => handOver(),++ onDrainDeadline: async () => await handOver(),onLost: () => stopAll('worker lost'),activeCount: () => runners.size,-- onExit: () => { closing = true; clearTimeout(pollTimer); if (runtime.me) store.outputs.release(runtime.me.id); },++ onExit: async () => { closing = true; clearTimeout(pollTimer); if (runtime.me) await store.outputs.release(runtime.me.id); },},});-- function inputUrlFor(session) {++ async function inputUrlFor(session) {if (session.protocol !== 'rtmp') return null;-- const pb = store.sessions.playback(session);++ const pb = await store.sessions.playback(session);return pb && pb.flv ? pb.flv.internal_url : null;}-- function poll() {++ async function poll() {if (closing || !runtime.me) return;-- const rows = store.outputs.forWorker(runtime.me.id);++ const rows = await store.outputs.forWorker(runtime.me.id);const seen = new Set();for (const row of rows) {seen.add(row.id);let runner = runners.get(row.id);-- const session = store.sessions.get(row.session_id);++ const session = await store.sessions.get(row.session_id);const shouldRun = row.desired === 'run' && session && session.state === 'live';if (!shouldRun) {-- if (runner) { runner.stop(row.desired === 'stop' ? 'stop requested' : 'session not live'); runners.delete(row.id); } else store.outputs.report(row.id, { state: 'stopped', ended_at: Date.now() }, { workerId: runtime.me.id });++ if (runner) { runner.stop(row.desired === 'stop' ? 'stop requested' : 'session not live'); runners.delete(row.id); } else await store.outputs.report(row.id, { state: 'stopped', ended_at: Date.now() }, { workerId: runtime.me.id });continue;}if (runner && runner.stopped) { runners.delete(row.id); runner = null; continue; }if (runner) continue;// A draining generation starts nothing new: an output assigned to it but not started yet// goes back to the coordinator for the newest generation.-- if (runtime.draining) { store.outputs.unassign(row.id); seen.delete(row.id); continue; }-- const input = inputUrlFor(session);-- if (!input) { store.outputs.report(row.id, { state: 'failed', last_error: `restream from ${session.protocol} sessions is not supported by OpenRe yet`, ended_at: Date.now() }, { workerId: runtime.me.id }); continue; }++ if (runtime.draining) { await store.outputs.unassign(row.id); seen.delete(row.id); continue; }++ const input = await inputUrlFor(session);++ if (!input) { await store.outputs.report(row.id, { state: 'failed', last_error: `restream from ${session.protocol} sessions is not supported by OpenRe yet`, ended_at: Date.now() }, { workerId: runtime.me.id }); continue; }// Let the ingest settle before pulling (Live waits 3 s for node-media-server's FLV).if (session.live_at && Date.now() - session.live_at < config.outputs.startDelayMs) continue;runner = new OutputRunner({ outputId: row.id, destinationId: row.destination_id, inputUrl: input, store, config, log, spawnImpl, lookup, workerId: runtime.me.id });@@@ -73,8 +73,8 @@ function createRestreamWorker({ rt, log = console, exit = (code) => process.exitfunction schedule() {clearTimeout(pollTimer);if (closing) return;-- pollTimer = setTimeout(() => {-- try { poll(); } catch (err) { log.error(`[restream] poll failed: ${err.stack || err}`); }++ pollTimer = setTimeout(async () => {++ try { await poll(); } catch (err) { log.error(`[restream] poll failed: ${err.stack || err}`); }schedule();}, config.workers.restreamPollMs);}@@@ -84,15 +84,15 @@ function createRestreamWorker({ rt, log = console, exit = (code) => process.exit}/** Drain deadline: stop here and let the coordinator give the outputs to the newest generation. */-- function handOver() {++ async function handOver() {for (const [id, r] of runners) { r.stopped = true; r.clearTimers(); if (r.proc) { try { r.proc.kill('SIGTERM'); } catch { /* */ } } runners.delete(id); }-- const n = store.outputs.release(runtime.me.id);++ const n = await store.outputs.release(runtime.me.id);log.log(`[restream] drain deadline: released ${n} output(s) to the newest generation`);}-- function start() {-- runtime.register({});-- runtime.ready();++ async function start() {++ await runtime.register({});++ await runtime.ready();schedule();log.log(`[restream] generation ${runtime.me.generation} ready`);return runtime.me;@@@ -103,26 +103,28 @@ function createRestreamWorker({ rt, log = console, exit = (code) => process.exitpoll,runtime,runners: () => runners,-- drain: (reason) => runtime.drainNow(reason),++ drain: async (reason) => await runtime.drainNow(reason),close() { closing = true; clearTimeout(pollTimer); stopAll('closing'); runtime.stop(); },};}if (require.main === module) {-- require('dotenv').config({ path: process.env.OPENRE_ENV_FILE || path.join(process.cwd(), '.env') });-- const { load, exitIfDrill } = require('../server/config');-- const { openRuntime } = require('../server/store');-- const config = load();-- exitIfDrill(config, 'openre-restream-worker');-- const rt = openRuntime({ config });-- const worker = createRestreamWorker({ rt });-- worker.start();-- // If this process dies, its ffmpeg children must not keep pushing unsupervised: the outputs are-- // reassigned to another worker, and two pushes to one ingest make platforms drop both.-- process.on('exit', () => {-- for (const r of worker.runners().values()) { try { if (r.proc) r.proc.kill('SIGKILL'); } catch { /* */ } }-- });-- for (const sig of ['SIGTERM', 'SIGINT']) process.on(sig, () => worker.drain(sig));++ (async () => {++ require('dotenv').config({ path: process.env.OPENRE_ENV_FILE || path.join(process.cwd(), '.env') });++ const { load, exitIfDrill } = require('../server/config');++ const { openRuntime } = require('../server/store');++ const config = load();++ exitIfDrill(config, 'openre-restream-worker');++ const rt = await openRuntime({ config });++ const worker = createRestreamWorker({ rt });++ worker.start();++ // If this process dies, its ffmpeg children must not keep pushing unsupervised: the outputs are++ // reassigned to another worker, and two pushes to one ingest make platforms drop both.++ process.on('exit', () => {++ for (const r of worker.runners().values()) { try { if (r.proc) r.proc.kill('SIGKILL'); } catch { /* */ } }++ });++ for (const sig of ['SIGTERM', 'SIGINT']) process.on(sig, () => worker.drain(sig));++ })().catch((err) => { console.error(`[restream-worker] failed to start: ${err && err.stack || err}`); process.exit(1); });}module.exports = { createRestreamWorker };@@@ -42,8 +42,8 @@ class OutputRunner {return t;}-- report(change) { try { return this.store.outputs.report(this.outputId, change, { workerId: this.workerId }); } catch (err) { this.log.error(`[restream] report ${this.outputId}: ${err.message}`); return null; } }-- note(level, message) { try { this.store.outputs.log(this.outputId, this.destinationId, level, this.scrub(message)); } catch { /* logs are best effort */ } }++ async report(change) { try { return await this.store.outputs.report(this.outputId, change, { workerId: this.workerId }); } catch (err) { this.log.error(`[restream] report ${this.outputId}: ${err.message}`); return null; } }++ async note(level, message) { try { await this.store.outputs.log(this.outputId, this.destinationId, level, this.scrub(message)); } catch { /* logs are best effort */ } }/** ffmpeg's stderr can echo the output URL: never let a destination key reach a log or event. */scrub(text) {@@@ -55,7 +55,7 @@ class OutputRunner {async start() {if (this.stopped) return;let dest;-- try { dest = this.store.outputs.destinationForWorker(this.destinationId); } catch (err) {++ try { dest = await this.store.outputs.destinationForWorker(this.destinationId); } catch (err) {return this.fail(`destination secrets unavailable: ${err.message}`, { cooldown: false });}if (!dest || !dest.enabled || dest.hold_reason) return this.stop('destination disabled');@@@ -139,13 +139,13 @@ class OutputRunner {}, this.o.liveAckTimeoutMs);}-- confirmLive(proc) {++ async confirmLive(proc) {if (this.proc !== proc || this.liveThisRun || this.stopped) return;this.liveThisRun = true;this.everLive = true;this.rapidCrashCount = 0;this.status = 'live';-- try { this.store.outputs.clearDestinationCooldown(this.destinationId); } catch { /* */ }++ try { await this.store.outputs.clearDestinationCooldown(this.destinationId); } catch { /* */ }this.report({ state: 'live', live_at: Date.now(), last_error: null });this.note('info', 'output is live');this.timer(() => {@@@ -201,14 +201,14 @@ class OutputRunner {}/** The circuit breaker: this output is done; the destination may cool down. */-- fail(message, { cooldown }) {++ async fail(message, { cooldown }) {if (this.stopped) return;message = this.scrub(message);this.stopped = true;this.clearTimers();let cooldownMinutes = null;if (cooldown) {-- const r = this.store.outputs.markDestinationFailure(this.destinationId, message);++ const r = await this.store.outputs.markDestinationFailure(this.destinationId, message);cooldownMinutes = r ? r.cooldownMinutes : null;}this.status = 'failed';@@@ -73,21 +73,22 @@ function createRtmpIngest({ rt, log = console, exit = (code) => process.exit(codrt, kind: 'rtmp-ingest', log, exit,hooks: {onDrain: () => closePublic(),-- onDrainDeadline: () => endAll('drain_deadline'),-- onEndRequested: (s) => endSession(s.id, 'end_requested'),-- onLost: () => { for (const [nmsId] of publishers) stopNms(nmsId); return closeAll(); },-- onHeartbeat: () => refreshMediaInfo(),++ onDrainDeadline: async () => await endAll('drain_deadline'),++ onEndRequested: async (s) => await endSession(s.id, 'end_requested'),++ onLost: async () => { for (const [nmsId] of publishers) await stopNms(nmsId); return closeAll(); },++ onHeartbeat: async () => await refreshMediaInfo(),activeCount: () => publishers.size,onExit: () => closeAll(),},});++ // node-media-server's own session registry (in memory), not the store.function nms(id) { return context.sessions.get(id); }function stopNms(id) { const s = nms(id); if (s) { try { s.reject(); } catch { /* socket already gone */ } } }function on(event, fn) {-- const wrapped = (...args) => {-- try { fn(...args); } catch (err) {++ const wrapped = async (...args) => {++ try { await fn(...args); } catch (err) {// A throw inside a node-media-server event is an uncaught exception, and that would// take every publisher on this worker with it.log.error(`[rtmp] ${event} handler failed: ${err.stack || err}`);@@@ -101,50 +102,84 @@ function createRtmpIngest({ rt, log = console, exit = (code) => process.exit(codreturn Boolean(session && session.socket && publicPorts.has(session.socket.localPort));}-- on('prePublish', (id, streamPath) => {++ /**++ * Admission of a publish: the key and the one-open-session rule are read from the store (PostgreSQL, async), but++ * node-media-server's onPublish is synchronous and registers the stream right after 'prePublish'. So every RTMP++ * session's publish command waits here first (newRtmpServer wraps onPublish), and 'prePublish' consumes the answer.++ */++ const admissions = new Map();++ async function admit(s, streamPath) {++ if (!isPublicSocket(s)) return { error: 'not the public listener' };++ if (runtime.draining || !runtime.me || runtime.me.state !== 'ready') return { error: 'generation not taking sessions' };++ if (publishers.size >= config.rtmp.maxPublishersPerWorker) return { error: 'worker full' };++ const parts = String(streamPath).split('/');++ if (parts.length !== 3 || parts[1] !== 'live') return { error: 'bad path' };++ const r = await store.definitions.resolveIngestKey(parts[2], 'rtmp');++ if (r.error) return { error: r.error };++ const a = await store.sessions.admit({ definition: r.definition, key: r.key, protocol: 'rtmp', worker: runtime.me });++ if (a.error) return { error: a.error };++ return { sessionId: a.session.id, definitionId: r.definition.id, hint: r.key.hint };++ }++ function admitThenPublish(session) {++ const onPublish = session.onPublish.bind(session);++ session.onPublish = (invokeMessage) => {++ // A second publish on a connection that already publishes: node-media-server answers++ // NetStream.Publish.BadConnection without touching the first stream.++ if (publishers.has(session.id)) return onPublish(invokeMessage);++ const streamPath = `/${session.appname}/${String(invokeMessage.streamName || '').split('?')[0]}`;++ admit(session, streamPath).then(async (adm) => {++ if (!session.isStarting) {++ // The publisher left while it was being admitted: its session ends without going live.++ if (adm.sessionId) await store.sessions.finish(adm.sessionId, { reason: 'publisher_disconnected', actor: `worker:${runtime.me.id}` });++ return;++ }++ admissions.set(session.id, adm);++ onPublish(invokeMessage);++ }, (err) => {++ log.error(`[rtmp] admission failed: ${err.stack || err}`);++ admissions.set(session.id, { error: 'admission failed' });++ if (session.isStarting) onPublish(invokeMessage);++ });++ };++ }++++ on('prePublish', (id) => {const s = nms(id);if (!s) return;const reject = (why) => { log.log(`[rtmp] publish refused (${why}) from ${s.ip}`); s.reject(); };-- // A second publish on a connection that already publishes: leave it to node-media-server,-- // which answers NetStream.Publish.BadConnection without touching the first stream.if (publishers.has(id)) return undefined;-- if (!isPublicSocket(s)) return reject('not the public listener');-- if (runtime.draining || !runtime.me || runtime.me.state !== 'ready') return reject('generation not taking sessions');-- if (publishers.size >= config.rtmp.maxPublishersPerWorker) return reject('worker full');-- const parts = String(streamPath).split('/');-- if (parts.length !== 3 || parts[1] !== 'live') return reject('bad path');-- const r = store.definitions.resolveIngestKey(parts[2], 'rtmp');-- if (r.error) return reject(r.error);-- const a = store.sessions.admit({ definition: r.definition, key: r.key, protocol: 'rtmp', worker: runtime.me });-- if (a.error) return reject(a.error);++ const adm = admissions.get(id);++ admissions.delete(id);++ if (!adm) return reject('not admitted');++ if (adm.error) return reject(adm.error);// Rename before node-media-server registers the publisher (it reads publishStreamPath// right after this synchronous event): from here on the stream is /live/<session id>.-- s.publishStreamPath = `/live/${a.session.id}`;-- publishers.set(id, { sessionId: a.session.id, definitionId: r.definition.id, endReason: null });-- log.log(`[rtmp] session ${a.session.id} accepted for ${r.definition.id} (key …${r.key.hint})`);++ s.publishStreamPath = `/live/${adm.sessionId}`;++ publishers.set(id, { sessionId: adm.sessionId, definitionId: adm.definitionId, endReason: null });++ log.log(`[rtmp] session ${adm.sessionId} accepted for ${adm.definitionId} (key …${adm.hint})`);});-- on('postPublish', (id) => {++ on('postPublish', async (id) => {const p = publishers.get(id);if (!p) return;-- const t = store.sessions.transition(p.sessionId, 'live', { reason: 'media_flowing', actor: `worker:${runtime.me.id}` });++ const t = await store.sessions.transition(p.sessionId, 'live', { reason: 'media_flowing', actor: `worker:${runtime.me.id}` });if (!t.ok) {// The coordinator failed it meanwhile (lease) — the transport must not outlive its record.log.warn(`[rtmp] session ${p.sessionId} could not go live (${t.code}); closing`);-- stopNms(id);++ await stopNms(id);}});-- on('donePublish', (id) => {++ on('donePublish', async (id) => {const p = publishers.get(id);if (!p) return;publishers.delete(id);-- store.sessions.finish(p.sessionId, { reason: p.endReason || 'publisher_disconnected', actor: `worker:${runtime.me.id}` });++ await store.sessions.finish(p.sessionId, { reason: p.endReason || 'publisher_disconnected', actor: `worker:${runtime.me.id}` });log.log(`[rtmp] session ${p.sessionId} ended (${p.endReason || 'publisher_disconnected'})`);if (runtime.draining) runtime.exitWhenIdle();});-- on('prePlay', (id, streamPath) => {++ on('prePlay', async (id, streamPath) => {const s = nms(id);if (!s) return;// RTMP play only on the loopback play port, and only by session path.@@@ -152,11 +187,11 @@ function createRtmpIngest({ rt, log = console, exit = (code) => process.exit(codif (!SESSION_PATH_RE.test(String(streamPath))) s.reject();});-- function refreshMediaInfo() {++ async function refreshMediaInfo() {for (const [id, p] of publishers) {const s = nms(id);if (!s || !s.isPublishing) continue;-- store.sessions.setMediaInfo(p.sessionId, {++ await store.sessions.setMediaInfo(p.sessionId, {video_codec: s.videoCodecName || null,width: s.videoWidth || null,height: s.videoHeight || null,@@@ -168,17 +203,17 @@ function createRtmpIngest({ rt, log = console, exit = (code) => process.exit(cod}}-- function endSession(sessionId, reason) {++ async function endSession(sessionId, reason) {for (const [id, p] of publishers) {-- if (p.sessionId === sessionId) { p.endReason = reason; stopNms(id); return true; }++ if (p.sessionId === sessionId) { p.endReason = reason; await stopNms(id); return true; }}// Not ours any more (publisher already gone): make sure the record is closed.-- store.sessions.finish(sessionId, { reason, actor: `worker:${runtime.me.id}` });++ await store.sessions.finish(sessionId, { reason, actor: `worker:${runtime.me.id}` });return false;}-- function endAll(reason) {-- for (const [id, p] of publishers) { p.endReason = reason; stopNms(id); }++ async function endAll(reason) {++ for (const [id, p] of publishers) { p.endReason = reason; await stopNms(id); }}function closePublic() {@@@ -209,6 +244,7 @@ function createRtmpIngest({ rt, log = console, exit = (code) => process.exit(codsrv.openSockets.add(socket);socket.on('close', () => srv.openSockets.delete(socket));const session = new NodeRtmpSession(nmsConfig, socket);++ admitThenPublish(session);session.run();});srv.openSockets = new Set();@@@ -233,7 +269,7 @@ function createRtmpIngest({ rt, log = console, exit = (code) => process.exit(cod});const flvPort = await listenInRange(flvServer, config.rtmp.internalPortMin, config.rtmp.internalPortMax, new Set([rtmpPlayPort]));endpoints = { publicPort: config.rtmp.port, publicPorts: [...publicPorts], rtmpPlayPort, flvPort };-- runtime.register(endpoints);++ await runtime.register(endpoints);for (const port of publicPorts) {const srv = newRtmpServer();@@@ -244,7 +280,7 @@ function createRtmpIngest({ rt, log = console, exit = (code) => process.exit(codsrv.on('error', (err) => log.error(`[rtmp] public listener ${port}: ${err.message}`));publicServers.push(srv);}-- runtime.ready();++ await runtime.ready();log.log(`[rtmp] generation ${runtime.me.generation} ready: publish ${config.rtmp.bindHost}:${[...publicPorts].join(',')}, play 127.0.0.1:${rtmpPlayPort}, flv 127.0.0.1:${flvPort}`);return endpoints;}@@@ -254,29 +290,31 @@ function createRtmpIngest({ rt, log = console, exit = (code) => process.exit(codruntime,endpoints: () => endpoints,publishers: () => [...publishers.values()],-- drain: (reason) => runtime.drainNow(reason),++ drain: async (reason) => await runtime.drainNow(reason),close: closeAll,};}if (require.main === module) {-- require('dotenv').config({ path: process.env.OPENRE_ENV_FILE || path.join(process.cwd(), '.env') });-- const { load, exitIfDrill } = require('../server/config');-- const { openRuntime } = require('../server/store');-- const config = load();-- exitIfDrill(config, 'openre-rtmp-ingest');-- const rt = openRuntime({ config });-- const worker = createRtmpIngest({ rt });-- let started = false;-- worker.start().then(() => { started = true; }, (err) => { console.error(`[rtmp] failed to start: ${err.stack || err}`); process.exit(1); });-- // node-media-server's own server logs an uncaught exception and keeps going (NodeMediaServer.run-- // registers exactly that). A malformed packet from one encoder must not take every other-- // publisher on this worker down with it, so the worker keeps that behaviour once it is up.-- process.on('uncaughtException', (err) => {-- console.error(`[rtmp] uncaught exception (worker keeps running): ${err && err.stack || err}`);-- if (!started) process.exit(1);-- });-- for (const sig of ['SIGTERM', 'SIGINT']) process.on(sig, () => worker.drain(sig));++ (async () => {++ require('dotenv').config({ path: process.env.OPENRE_ENV_FILE || path.join(process.cwd(), '.env') });++ const { load, exitIfDrill } = require('../server/config');++ const { openRuntime } = require('../server/store');++ const config = load();++ exitIfDrill(config, 'openre-rtmp-ingest');++ const rt = await openRuntime({ config });++ const worker = createRtmpIngest({ rt });++ let started = false;++ worker.start().then(() => { started = true; }, (err) => { console.error(`[rtmp] failed to start: ${err.stack || err}`); process.exit(1); });++ // node-media-server's own server logs an uncaught exception and keeps going (NodeMediaServer.run++ // registers exactly that). A malformed packet from one encoder must not take every other++ // publisher on this worker down with it, so the worker keeps that behaviour once it is up.++ process.on('uncaughtException', (err) => {++ console.error(`[rtmp] uncaught exception (worker keeps running): ${err && err.stack || err}`);++ if (!started) process.exit(1);++ });++ for (const sig of ['SIGTERM', 'SIGINT']) process.on(sig, () => worker.drain(sig));++ })().catch((err) => { console.error(`[rtmp-ingest] failed to start: ${err && err.stack || err}`); process.exit(1); });}module.exports = { createRtmpIngest, listenInRange };@@@ -37,22 +37,22 @@ function createWorkerRuntime({ rt, kind, log = console, hooks = {}, exit = (codelet deadlineHandled = false;let stopped = false;-- function register(endpoints = {}) {-- me = store.workers.register({ kind, endpoints });++ async function register(endpoints = {}) {++ me = await store.workers.register({ kind, endpoints });log.log(`[${kind}] registered ${me.id} generation ${me.generation} (release ${me.release})`);return me;}-- function ready() {-- store.workers.ready(me.id);-- me = store.workers.get(me.id);++ async function ready() {++ await store.workers.ready(me.id);++ me = await store.workers.get(me.id);schedule();return me;}-- function beat() {++ async function beat() {if (stopped || !me) return null;-- const row = store.workers.heartbeat(me.id);++ const row = await store.workers.heartbeat(me.id);me = row;if (!row) return null;if (row.state === 'lost') {@@@ -73,7 +73,7 @@ function createWorkerRuntime({ rt, kind, log = console, hooks = {}, exit = (codetry { hooks.onDrainDeadline && hooks.onDrainDeadline(row); } catch (err) { log.error(`[${kind}] onDrainDeadline: ${err.message}`); }}if (hooks.onEndRequested) {-- for (const s of store.sessions.endRequestsFor(me.id)) {++ for (const s of await store.sessions.endRequestsFor(me.id)) {try { hooks.onEndRequested(s); } catch (err) { log.error(`[${kind}] onEndRequested: ${err.message}`); }}}@@@ -87,20 +87,20 @@ function createWorkerRuntime({ rt, kind, log = console, hooks = {}, exit = (codefunction schedule() {clearTimeout(timer);if (stopped) return;-- timer = setTimeout(() => {-- try { beat(); } catch (err) { log.error(`[${kind}] heartbeat failed: ${err.message}`); }++ timer = setTimeout(async () => {++ try { await beat(); } catch (err) { log.error(`[${kind}] heartbeat failed: ${err.message}`); }schedule();}, config.workers.heartbeatMs);}/** SIGTERM: the operator (or systemd) asks this generation to go away → same as a drain. */-- function drainNow(reason = 'signal') {++ async function drainNow(reason = 'signal') {if (!me || stopped) return;if (!draining) {-- store.workers.drain(me.id, Date.now() + config.workers.drainMaxMs);++ await store.workers.drain(me.id, Date.now() + config.workers.drainMaxMs);log.log(`[${kind}] draining on ${reason}`);}-- beat();++ await beat();}function exitWhenIdle() {@@@ -1,3 +1,3 @@'use strict';/** openre-sfu: worker interface only, the transport is not ported yet (see unported-transport.js). */--if (require.main === module) require('./unported-transport').main('sfu');++if (require.main === module) require('./unported-transport').main('sfu').catch((err) => { console.error(`[sfu] failed to start: ${err && err.stack || err}`); process.exit(1); });@@@ -24,23 +24,23 @@ function createUnportedTransport({ rt, kind, log = console, exit = (code) => proconst runtime = createWorkerRuntime({ rt, kind, log, exit, hooks: { activeCount: () => 0 } });return {runtime,-- start() {-- runtime.register({ ported: false, note: PORT_STATUS[kind] });-- runtime.ready();++ async start() {++ await runtime.register({ ported: false, note: PORT_STATUS[kind] });++ await runtime.ready();log.log(`[${kind}] registered generation ${runtime.me.generation}: ${PORT_STATUS[kind]}`);return runtime.me;},-- drain: (reason) => runtime.drainNow(reason),++ drain: async (reason) => await runtime.drainNow(reason),};}--function main(kind) {++async function main(kind) {require('dotenv').config({ path: process.env.OPENRE_ENV_FILE || path.join(process.cwd(), '.env') });const { load, exitIfDrill } = require('../server/config');const { openRuntime } = require('../server/store');const config = load();exitIfDrill(config, `openre-${kind}`);-- const w = createUnportedTransport({ rt: openRuntime({ config }), kind });++ const w = createUnportedTransport({ rt: await openRuntime({ config }), kind });w.start();for (const sig of ['SIGTERM', 'SIGINT']) process.on(sig, () => w.drain(sig));}@@@ -1,3 +1,3 @@'use strict';/** openre-webrtc-ingest: worker interface only, the transport is not ported yet (see unported-transport.js). */--if (require.main === module) require('./unported-transport').main('webrtc-ingest');++if (require.main === module) require('./unported-transport').main('webrtc-ingest').catch((err) => { console.error(`[webrtc-ingest] failed to start: ${err && err.stack || err}`); process.exit(1); });
@@ -30,6 +30,13 @@ OV_OAUTH_CLIENT_SECRET=
@@ -30,6 +30,13 @@ OV_OAUTH_CLIENT_SECRET=
#OPENRE_USER_AUDIENCES=openvibe.openre,openvibe.network
#OPENRE_USER_AUDIENCES=openvibe.openre,openvibe.network
# ── Store (SQLite, shared by every OpenRe process; WAL) ───────
# ── Store (SQLite, shared by every OpenRe process; WAL) ───────
+# PostgreSQL (ADR-035): DATABASE_URL serves through PgBouncer, DATABASE_DIRECT_URL (the owner) runs migrations/ at boot;
+# OpenVibe.Host `roles/data/add-service.sh openre` writes both. Every OpenRe process shares the database. Without
+# DATABASE_URL, development runs on an embedded PGlite database in data/pglite (OPENRE_PGLITE_DIR): one process only.
+DATABASE_URL=
+DATABASE_DIRECT_URL=
+VALKEY_URL=
+# The SQLite file the one-time move reads (scripts/migrate-to-postgres.js).
OPENRE_DB_PATH=./data/openre.db
OPENRE_DB_PATH=./data/openre.db
# ── Secrets ───────────────────────────────────────────────────
# ── Secrets ───────────────────────────────────────────────────
@@ -4,11 +4,14 @@ on:
@@ -4,11 +4,14 @@ on:
branches: [main]
branches: [main]
pull_request:
pull_request:
jobs:
jobs:
- # install, node --check, npm test and openvibe-contracts-check: the shared test job (Track Q).
+ # install, node --check, npm test and openvibe-contracts-check: the shared test job (Track Q). ADR-035: PostgreSQL
+ # 18 + PgBouncer (transaction mode) + Valkey 9 containers (openvibe-sdk scripts/test-services.sh); npm test runs every
+ # file on PGlite, test:pg runs them all through PgBouncer, with the tests that start several OpenRe processes.
test:
test:
- uses: OpenVibers/OpenVibe.Shared/.github/workflows/test.yml@e8b0caf086ad1f8fceaac80e2dca6cfe64f63602
+ uses: OpenVibers/OpenVibe.Shared/.github/workflows/test.yml@423fe519b5fe9594cc81b670e5ba5cb9f21f6176
with:
with:
apt-packages: ffmpeg # the RTMP end-to-end test publishes and restreams with a real ffmpeg
apt-packages: ffmpeg # the RTMP end-to-end test publishes and restreams with a real ffmpeg
+ test: 'eval "$(node_modules/openvibe-sdk/scripts/test-services.sh up)" && npm test && npm run test:pg'
syntax-dirs: server scripts workers
syntax-dirs: server scripts workers
contracts-service: openre
contracts-service: openre
contracts-src: server workers
contracts-src: server workers
@@ -32,7 +32,8 @@ The ingest/restream control plane and runtime extracted from OpenVibe.Live. Live
@@ -32,7 +32,8 @@ The ingest/restream control plane and runtime extracted from OpenVibe.Live. Live
- OpenVibe.Network (JWKS for service tokens and user JWTs; OAuth client `openre`; client-credentials tokens for Events/Media)
- OpenVibe.Network (JWKS for service tokens and user JWTs; OAuth client `openre`; client-credentials tokens for Events/Media)
- OpenVibe.Events (event relay; optional: rows wait in the outbox)
- OpenVibe.Events (event relay; optional: rows wait in the outbox)
- OpenVibe.Media (recording requests; optional: sessions work without it)
- OpenVibe.Media (recording requests; optional: sessions work without it)
-- OpenVibe.Contracts v0.49.0 (ids, problem+json, capability checks), OpenVibe.SDK v0.12.0 (outbox, token client, per-actor limits), OpenVibe.Shared v1.25.0 (chrome, readiness), pinned by release tarball
+- OpenVibe.Contracts v0.76.0 (ids, problem+json, capability checks), OpenVibe.SDK v0.21.3 (`db`, PostgreSQL outbox, token client, per-actor limits), OpenVibe.Shared v1.28.0 (Frame, readiness), pinned by release tarball
+- **PostgreSQL 18 and Valkey 9** (OpenVibe.Host `roles/data/`, ADR-035): the store every OpenRe process shares, and the API's per-actor limit counters (optional)
- OpenVibe.Live (`live.lineage.resolve`: which channel a Live-linked stream belongs to)
- OpenVibe.Live (`live.lineage.resolve`: which channel a Live-linked stream belongs to)
- node-media-server 2.7.4 (the RTMP session code Live runs) and the system ffmpeg
- node-media-server 2.7.4 (the RTMP session code Live runs) and the system ffmpeg
@@ -53,7 +54,7 @@ The ingest/restream control plane and runtime extracted from OpenVibe.Live. Live
@@ -53,7 +54,7 @@ The ingest/restream control plane and runtime extracted from OpenVibe.Live. Live
## Processes
## Processes
```
```
- ┌───────────────── one SQLite database (WAL), /var/lib/openre/openre.db ─────────────────┐
+ ┌───────────────── one PostgreSQL database, ov_openre (ADR-035), through PgBouncer ─────────────────┐
openre-api (4500) openre-session-coordinator openre-rtmp-ingest@<release> openre-restream-worker@<release>
openre-api (4500) openre-session-coordinator openre-rtmp-ingest@<release> openre-restream-worker@<release>
API + UI + /play proxy leases, generations, drain, 0.0.0.0:1936 publish (reuseport) one ffmpeg per output, pulls the
API + UI + /play proxy leases, generations, drain, 0.0.0.0:1936 publish (reuseport) one ffmpeg per output, pulls the
definitions, keys, output assignment, recording 127.0.0.1 play + HTTP-FLV session's HTTP-FLV from its worker
definitions, keys, output assignment, recording 127.0.0.1 play + HTTP-FLV session's HTTP-FLV from its worker
@@ -64,7 +65,7 @@ openre-api (4500) openre-session-coordinator openre-rtmp-inges
@@ -64,7 +65,7 @@ openre-api (4500) openre-session-coordinator openre-rtmp-inges
- **Generations.** Each worker process registers as generation N+1 of its kind. When a newer generation is `ready`, the coordinator marks older ones `draining` (deadline `OPENRE_DRAIN_MAX_MS`, default 24 h). A draining RTMP worker closes its public listener (the kernel sends new encoder connections to the new generation through the shared port) and keeps its publishers; a draining restream worker keeps its outputs. Idle → the process exits 0 by itself. At the deadline, remaining sessions are ended (encoders reconnect to the newest generation) and remaining outputs are handed over.
- **Generations.** Each worker process registers as generation N+1 of its kind. When a newer generation is `ready`, the coordinator marks older ones `draining` (deadline `OPENRE_DRAIN_MAX_MS`, default 24 h). A draining RTMP worker closes its public listener (the kernel sends new encoder connections to the new generation through the shared port) and keeps its publishers; a draining restream worker keeps its outputs. Idle → the process exits 0 by itself. At the deadline, remaining sessions are ended (encoders reconnect to the newest generation) and remaining outputs are handed over.
- **Leases.** Every heartbeat (2 s) renews the worker and the lease of every session it owns in one transaction. A worker silent for `OPENRE_WORKER_LEASE_MS` (15 s) is `lost`: its sessions fail (`openre.session.failed`), its outputs go back to `pending` for another worker. A lost worker that wakes up drops its transports and exits.
- **Leases.** Every heartbeat (2 s) renews the worker and the lease of every session it owns in one transaction. A worker silent for `OPENRE_WORKER_LEASE_MS` (15 s) is `lost`: its sessions fail (`openre.session.failed`), its outputs go back to `pending` for another worker. A lost worker that wakes up drops its transports and exits.
- **Isolation.** An output only ever changes its own row. A dead destination fails (`openre.output.failed`, destination cooldown 15 min → 24 h) and the session stays live (`test/rtmp-e2e.test.js`, `test/sessions.test.js`, `test/coordinator.test.js`).
- **Isolation.** An output only ever changes its own row. A dead destination fails (`openre.output.failed`, destination cooldown 15 min → 24 h) and the session stays live (`test/rtmp-e2e.test.js`, `test/sessions.test.js`, `test/coordinator.test.js`).
-- **Store.** SQLite per ADR-007 (single host). Several writer processes is the trigger ADR-007 names for PostgreSQL; all SQL is in `server/store/*`, so that move replaces one layer. Redis/etcd are not used; leases live in the same database.
+- **Store.** PostgreSQL 18 through PgBouncer (`ov_openre` on the host's data role, ADR-035; schema in [migrations/](migrations/)), shared by every OpenRe process. Every transaction is SERIALIZABLE (`server/store/index.js`): an ingest worker admitting a publish and the coordinator failing a lost worker's session conflict, and PostgreSQL makes one of them retry, as SQLite's one writer ordered them. Admission is async, so the RTMP worker admits a publish before node-media-server's own publish handling runs. Valkey holds the API's per-actor limit counters. The one-time move is `scripts/migrate-to-postgres.js` (run by OpenVibe.Host `roles/data/switch-service.sh` with every OpenRe unit stopped); `/var/lib/openre/openre.db` stays read-only for 7 days as the rollback.
## Running it
## Running it
@@ -124,7 +125,7 @@ The ids were proposed in [docs/capabilities-proposal/](docs/capabilities-proposa
@@ -124,7 +125,7 @@ The ids were proposed in [docs/capabilities-proposal/](docs/capabilities-proposa
## Events
## Events
-Written to `event_outbox` in the same SQLite transaction as the change (openvibe-sdk `createOutbox`); the coordinator's relay publishes them with OpenRe's service token. Envelopes validate against `events.event-envelope@1`; `visibility: internal`; no payload ever carries an ingest key or a destination key.
+Written to `event_outbox` in the same transaction as the change (openvibe-sdk `createPgOutbox`); the coordinator's relay publishes them with OpenRe's service token. Envelopes validate against `events.event-envelope@1`; `visibility: internal`; no payload ever carries an ingest key or a destination key.
| Type | Subject | When |
| Type | Subject | When |
|---|---|---|
|---|---|---|
@@ -167,7 +168,7 @@ The switch to 1935: once Live no longer listens on 1935, set `OPENRE_RTMP_EXTRA_
@@ -167,7 +168,7 @@ The switch to 1935: once Live no longer listens on 1935, set `OPENRE_RTMP_EXTRA_
## Deploying
## Deploying
-Layout: `/opt/openre.stream/repo` (the git clone releases are made from), `/opt/openre.stream/releases/<sha12>` (full checkouts with their own `node_modules`, owned by ubuntu), `current` → the release the API and coordinator run, env `/etc/openvibe/openre.env` (see [.env.example](.env.example)), store `/var/lib/openre/openre.db`, units in [deploy/systemd/](deploy/systemd/), nginx [deploy/nginx/openre.stream.conf](deploy/nginx/openre.stream.conf), script [deploy/scripts/deploy.sh](deploy/scripts/deploy.sh). The release and API phases run through `ovhost deploy openre` (OpenVibe.Host, strategy `release-layout`; roadmap WS-N task 11); the script is a thin wrapper that keeps the old subcommands:
+Layout: `/opt/openre.stream/repo` (the git clone releases are made from), `/opt/openre.stream/releases/<sha12>` (full checkouts with their own `node_modules`, owned by ubuntu), `current` → the release the API and coordinator run, env `/etc/openvibe/openre.env` (see [.env.example](.env.example)), store `ov_openre` on PostgreSQL (`sudo /opt/openvibe.host/roles/data/add-service.sh openre` writes its settings; the release migrates it at boot; the old SQLite file is `/var/lib/openre/openre.db`), units in [deploy/systemd/](deploy/systemd/), nginx [deploy/nginx/openre.stream.conf](deploy/nginx/openre.stream.conf), script [deploy/scripts/deploy.sh](deploy/scripts/deploy.sh). The release and API phases run through `ovhost deploy openre` (OpenVibe.Host, strategy `release-layout`; roadmap WS-N task 11); the script is a thin wrapper that keeps the old subcommands:
```bash
```bash
sudo deploy/scripts/deploy.sh # ovhost deploy openre: release + api in one
sudo deploy/scripts/deploy.sh # ovhost deploy openre: release + api in one
@@ -4,11 +4,11 @@
@@ -4,11 +4,11 @@
"code": true,
"code": true,
"wave": "W7",
"wave": "W7",
"plan": "OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026)",
"plan": "OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026)",
- "runtime": "node 22, express, better-sqlite3, node-media-server 2.7.4, ffmpeg",
+ "runtime": "node 22, express, PostgreSQL 18 (openvibe-sdk/db, async) + Valkey 9, node-media-server 2.7.4, ffmpeg",
"port": 4500,
"port": 4500,
"rtmpPort": 1936,
"rtmpPort": 1936,
"domain": "openre.stream",
"domain": "openre.stream",
- "contracts": "openvibe-contracts v0.49.0 (openvibe-sdk v0.12.0, openvibe-shared v1.25.0)",
+ "contracts": "openvibe-contracts v0.76.0 (openvibe-sdk v0.21.3, openvibe-shared v1.28.0)",
"protocols": {
"protocols": {
"rtmp": "ingest ported (openre-rtmp-ingest)",
"rtmp": "ingest ported (openre-rtmp-ingest)",
"restream": "RTMP source to RTMP/RTMPS/SRT destinations ported (openre-restream-worker)",
"restream": "RTMP source to RTMP/RTMPS/SRT destinations ported (openre-restream-worker)",
@@ -16,7 +16,15 @@
@@ -16,7 +16,15 @@
"webrtc-sfu": "not ported (worker interface only)",
"webrtc-sfu": "not ported (worker interface only)",
"jsmpeg": "not ported (worker interface only)"
"jsmpeg": "not ported (worker interface only)"
},
},
- "capabilities": ["openre.stream.read", "openre.stream.write", "openre.key.rotate", "openre.session.read", "openre.session.end", "openre.output.read", "openre.output.write"],
+ "capabilities": [
+ "openre.stream.read",
+ "openre.stream.write",
+ "openre.key.rotate",
+ "openre.session.read",
+ "openre.session.end",
+ "openre.output.read",
+ "openre.output.write"
+ ],
"capabilitiesRegistered": true,
"capabilitiesRegistered": true,
"livePatch": "applied in Live f0ca18b and deployed; Live wired on 2026-09-23 (OPENRE_URL and the openre.session.* subscription, docs/cutover.md phase A); every Live slot keeps ingest_authority=live until its per-slot cutover (phase B), so no broadcaster uses OpenRe yet",
"livePatch": "applied in Live f0ca18b and deployed; Live wired on 2026-09-23 (OPENRE_URL and the openre.session.* subscription, docs/cutover.md phase A); every Live slot keeps ingest_authority=live until its per-slot cutover (phase B), so no broadcaster uses OpenRe yet",
"deployed": true,
"deployed": true,
@@ -2,8 +2,8 @@
@@ -2,8 +2,8 @@
Description=OpenRe.Stream API and UI (openre-api): definitions, keys, destinations, sessions, playback proxy
Description=OpenRe.Stream API and UI (openre-api): definitions, keys, destinations, sessions, playback proxy
# The API owns no transport. Restarting it (every API deploy) never touches the transport
# The API owns no transport. Restarting it (every API deploy) never touches the transport
# workers: they are separate units with no dependency on this one, in either direction.
# workers: they are separate units with no dependency on this one, in either direction.
-After=network.target
-Wants=openvibe-network.service
+After=network.target postgresql.service pgbouncer.service valkey-server.service
+Wants=openvibe-network.service postgresql.service pgbouncer.service valkey-server.service
[Service]
[Service]
Type=simple
Type=simple
@@ -13,6 +13,7 @@ WorkingDirectory=/opt/openre.stream/current
@@ -13,6 +13,7 @@ WorkingDirectory=/opt/openre.stream/current
EnvironmentFile=/etc/openvibe/openre.env
EnvironmentFile=/etc/openvibe/openre.env
Environment=NODE_ENV=production
Environment=NODE_ENV=production
Environment=PORT=4500
Environment=PORT=4500
+# The SQLite file the one-time move read (scripts/migrate-to-postgres.js); the store is ov_openre (ADR-035).
Environment=OPENRE_DB_PATH=/var/lib/openre/openre.db
Environment=OPENRE_DB_PATH=/var/lib/openre/openre.db
Environment=PATH=/usr/local/bin:/usr/bin:/bin
Environment=PATH=/usr/local/bin:/usr/bin:/bin
ExecStart=/usr/bin/env node server/index.js
ExecStart=/usr/bin/env node server/index.js
@@ -5,7 +5,8 @@ Description=OpenRe.Stream restream worker (ffmpeg per destination), release %i
@@ -5,7 +5,8 @@ Description=OpenRe.Stream restream worker (ffmpeg per destination), release %i
# outputs run until their sessions end (or OPENRE_DRAIN_MAX_MS: then they move over) before it exits 0.
# outputs run until their sessions end (or OPENRE_DRAIN_MAX_MS: then they move over) before it exits 0.
# Nothing here depends on openre-api or openre-session-coordinator: an API deploy never restarts
# Nothing here depends on openre-api or openre-session-coordinator: an API deploy never restarts
# this unit. Never `systemctl restart` it during a broadcast; see README "Deploying".
# this unit. Never `systemctl restart` it during a broadcast; see README "Deploying".
-After=network.target
+After=network.target postgresql.service pgbouncer.service valkey-server.service
+Wants=postgresql.service pgbouncer.service valkey-server.service
[Service]
[Service]
Type=simple
Type=simple
@@ -15,6 +16,7 @@ WorkingDirectory=/opt/openre.stream/releases/%i
@@ -15,6 +16,7 @@ WorkingDirectory=/opt/openre.stream/releases/%i
EnvironmentFile=/etc/openvibe/openre.env
EnvironmentFile=/etc/openvibe/openre.env
Environment=NODE_ENV=production
Environment=NODE_ENV=production
Environment=OPENRE_RELEASE=%i
Environment=OPENRE_RELEASE=%i
+# The SQLite file the one-time move read (scripts/migrate-to-postgres.js); the store is ov_openre (ADR-035).
Environment=OPENRE_DB_PATH=/var/lib/openre/openre.db
Environment=OPENRE_DB_PATH=/var/lib/openre/openre.db
Environment=PATH=/usr/local/bin:/usr/bin:/bin
Environment=PATH=/usr/local/bin:/usr/bin:/bin
ExecStart=/usr/bin/env node workers/restream-worker.js
ExecStart=/usr/bin/env node workers/restream-worker.js
@@ -5,7 +5,8 @@ Description=OpenRe.Stream RTMP ingest worker, release %i
@@ -5,7 +5,8 @@ Description=OpenRe.Stream RTMP ingest worker, release %i
# keeps its publishers until they leave (or OPENRE_DRAIN_MAX_MS), and exits 0 by itself.
# keeps its publishers until they leave (or OPENRE_DRAIN_MAX_MS), and exits 0 by itself.
# Nothing here depends on openre-api or openre-session-coordinator: an API deploy never restarts
# Nothing here depends on openre-api or openre-session-coordinator: an API deploy never restarts
# this unit. Never `systemctl restart` it during a broadcast; see README "Deploying".
# this unit. Never `systemctl restart` it during a broadcast; see README "Deploying".
-After=network.target
+After=network.target postgresql.service pgbouncer.service valkey-server.service
+Wants=postgresql.service pgbouncer.service valkey-server.service
[Service]
[Service]
Type=simple
Type=simple
@@ -15,6 +16,7 @@ WorkingDirectory=/opt/openre.stream/releases/%i
@@ -15,6 +16,7 @@ WorkingDirectory=/opt/openre.stream/releases/%i
EnvironmentFile=/etc/openvibe/openre.env
EnvironmentFile=/etc/openvibe/openre.env
Environment=NODE_ENV=production
Environment=NODE_ENV=production
Environment=OPENRE_RELEASE=%i
Environment=OPENRE_RELEASE=%i
+# The SQLite file the one-time move read (scripts/migrate-to-postgres.js); the store is ov_openre (ADR-035).
Environment=OPENRE_DB_PATH=/var/lib/openre/openre.db
Environment=OPENRE_DB_PATH=/var/lib/openre/openre.db
Environment=PATH=/usr/local/bin:/usr/bin:/bin
Environment=PATH=/usr/local/bin:/usr/bin:/bin
ExecStart=/usr/bin/env node workers/rtmp-ingest.js
ExecStart=/usr/bin/env node workers/rtmp-ingest.js
@@ -1,7 +1,8 @@
@@ -1,7 +1,8 @@
[Unit]
[Unit]
Description=OpenRe.Stream session coordinator: worker leases and generations, output assignment, recording requests, event relay
Description=OpenRe.Stream session coordinator: worker leases and generations, output assignment, recording requests, event relay
# Stateless between ticks (everything is in the database): restarting it never ends a session.
# Stateless between ticks (everything is in the database): restarting it never ends a session.
-After=network.target
+After=network.target postgresql.service pgbouncer.service valkey-server.service
+Wants=postgresql.service pgbouncer.service valkey-server.service
[Service]
[Service]
Type=simple
Type=simple
@@ -10,6 +11,7 @@ Group=ubuntu
@@ -10,6 +11,7 @@ Group=ubuntu
WorkingDirectory=/opt/openre.stream/current
WorkingDirectory=/opt/openre.stream/current
EnvironmentFile=/etc/openvibe/openre.env
EnvironmentFile=/etc/openvibe/openre.env
Environment=NODE_ENV=production
Environment=NODE_ENV=production
+# The SQLite file the one-time move read (scripts/migrate-to-postgres.js); the store is ov_openre (ADR-035).
Environment=OPENRE_DB_PATH=/var/lib/openre/openre.db
Environment=OPENRE_DB_PATH=/var/lib/openre/openre.db
Environment=PATH=/usr/local/bin:/usr/bin:/bin
Environment=PATH=/usr/local/bin:/usr/bin:/bin
ExecStart=/usr/bin/env node workers/coordinator.js
ExecStart=/usr/bin/env node workers/coordinator.js
new file mode 100644
new file mode 100644
@@ -0,0 +1,221 @@
@@ -0,0 +1,221 @@
+-- phase: expand
+-- OpenVibe.OpenRe on PostgreSQL (ADR-035, roadmap WS-X2): the tables as they were on SQLite (converted by openvibe-sdk
+-- tools/asyncify/sqlite-schema-to-pg: text COLLATE "C" compares like SQLite, integers are bigint, identities keep their ids),
+-- then the openvibe-publishing stores and the openvibe-sdk inbox and outbox. Generated once on 2026-09-28; never edited after it runs.
+CREATE TABLE stream_definitions (
+ id text COLLATE "C" PRIMARY KEY,
+ owner_subject text COLLATE "C" NOT NULL,
+ title text COLLATE "C" NOT NULL DEFAULT 'Untitled stream',
+ description text COLLATE "C" NOT NULL DEFAULT '',
+ protocols text COLLATE "C" NOT NULL DEFAULT '["rtmp"]',
+ recording_mode text COLLATE "C" NOT NULL DEFAULT 'vod' CHECK (recording_mode IN ('vod', 'clips', 'none')),
+ recording_visibility text COLLATE "C" NOT NULL DEFAULT 'public' CHECK (recording_visibility IN ('public', 'unlisted', 'private')),
+ playback_visibility text COLLATE "C" NOT NULL DEFAULT 'public' CHECK (playback_visibility IN ('public', 'unlisted', 'private')),
+ mirror_to_live bigint NOT NULL DEFAULT 0,
+ state text COLLATE "C" NOT NULL DEFAULT 'active' CHECK (state IN ('active', 'disabled', 'archived')),
+ revision bigint NOT NULL DEFAULT 1,
+ created_at bigint NOT NULL,
+ updated_at bigint NOT NULL
+);
+
+CREATE TABLE external_refs (
+ definition_id text COLLATE "C" NOT NULL REFERENCES stream_definitions(id) ON DELETE CASCADE,
+ service text COLLATE "C" NOT NULL,
+ type text COLLATE "C" NOT NULL,
+ ref_id text COLLATE "C" NOT NULL,
+ label text COLLATE "C",
+ created_at bigint NOT NULL,
+ PRIMARY KEY (definition_id, service, type, ref_id)
+);
+
+CREATE TABLE definition_lineage (
+ definition_id text COLLATE "C" PRIMARY KEY REFERENCES stream_definitions(id) ON DELETE CASCADE,
+ resolution text COLLATE "C" NOT NULL,
+ checked_at bigint NOT NULL
+);
+
+CREATE TABLE ingest_keys (
+ id text COLLATE "C" PRIMARY KEY,
+ definition_id text COLLATE "C" NOT NULL REFERENCES stream_definitions(id) ON DELETE CASCADE,
+ key_hash text COLLATE "C" NOT NULL UNIQUE,
+ hint text COLLATE "C" NOT NULL,
+ status text COLLATE "C" NOT NULL CHECK (status IN ('active', 'grace', 'revoked')),
+ created_by text COLLATE "C",
+ created_at bigint NOT NULL,
+ grace_until bigint,
+ revoked_at bigint,
+ revoked_reason text COLLATE "C",
+ last_used_at bigint
+);
+
+CREATE TABLE workers (
+ id text COLLATE "C" PRIMARY KEY,
+ kind text COLLATE "C" NOT NULL,
+ generation bigint NOT NULL,
+ release text COLLATE "C",
+ pid bigint,
+ host text COLLATE "C",
+ state text COLLATE "C" NOT NULL CHECK (state IN ('starting', 'ready', 'draining', 'stopped', 'lost')),
+ endpoints text COLLATE "C" NOT NULL DEFAULT '{}',
+ started_at bigint NOT NULL,
+ ready_at bigint,
+ heartbeat_at bigint NOT NULL,
+ drain_started_at bigint,
+ drain_deadline bigint,
+ stopped_at bigint,
+ stop_reason text COLLATE "C",
+ UNIQUE (kind, generation)
+);
+
+CREATE TABLE ingest_sessions (
+ id text COLLATE "C" PRIMARY KEY,
+ definition_id text COLLATE "C" NOT NULL REFERENCES stream_definitions(id),
+ key_id text COLLATE "C",
+ protocol text COLLATE "C" NOT NULL,
+ state text COLLATE "C" NOT NULL CHECK (state IN ('starting', 'live', 'ending', 'ended', 'failed')),
+ worker_id text COLLATE "C",
+ worker_kind text COLLATE "C",
+ worker_generation bigint,
+ lease_expires_at bigint,
+ desired_state text COLLATE "C" NOT NULL DEFAULT 'run' CHECK (desired_state IN ('run', 'end')),
+ end_requested_by text COLLATE "C",
+ end_reason text COLLATE "C",
+ failure_reason text COLLATE "C",
+ media_info text COLLATE "C",
+ revision bigint NOT NULL DEFAULT 1,
+ created_at bigint NOT NULL,
+ live_at bigint,
+ ending_at bigint,
+ ended_at bigint,
+ updated_at bigint NOT NULL
+);
+
+CREATE TABLE session_transitions (
+ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
+ session_id text COLLATE "C" NOT NULL REFERENCES ingest_sessions(id) ON DELETE CASCADE,
+ from_state text COLLATE "C",
+ to_state text COLLATE "C" NOT NULL,
+ reason text COLLATE "C",
+ actor text COLLATE "C",
+ at bigint NOT NULL
+);
+
+CREATE TABLE destinations (
+ id text COLLATE "C" PRIMARY KEY,
+ definition_id text COLLATE "C" NOT NULL REFERENCES stream_definitions(id) ON DELETE CASCADE,
+ platform text COLLATE "C" NOT NULL CHECK (platform IN ('youtube', 'twitch', 'kick', 'custom')),
+ name text COLLATE "C",
+ server_url text COLLATE "C" NOT NULL,
+ stream_key_enc text COLLATE "C",
+ key_hint text COLLATE "C",
+ srt_passphrase_enc text COLLATE "C",
+ srt_latency_ms bigint,
+ enabled bigint NOT NULL DEFAULT 1,
+ auto_start bigint NOT NULL DEFAULT 1,
+ quality_preset text COLLATE "C" NOT NULL DEFAULT 'auto',
+ custom_video_bitrate bigint,
+ custom_audio_bitrate bigint,
+ custom_fps bigint,
+ custom_encoder_preset text COLLATE "C",
+ hold_reason text COLLATE "C",
+ consecutive_failures bigint NOT NULL DEFAULT 0,
+ cooldown_until bigint,
+ last_error text COLLATE "C",
+ last_failed_at bigint,
+ created_at bigint NOT NULL,
+ updated_at bigint NOT NULL
+);
+
+CREATE TABLE outputs (
+ id text COLLATE "C" PRIMARY KEY,
+ session_id text COLLATE "C" NOT NULL REFERENCES ingest_sessions(id),
+ destination_id text COLLATE "C" NOT NULL REFERENCES destinations(id) ON DELETE CASCADE,
+ desired text COLLATE "C" NOT NULL DEFAULT 'run' CHECK (desired IN ('run', 'stop')),
+ state text COLLATE "C" NOT NULL DEFAULT 'pending' CHECK (state IN ('pending', 'starting', 'live', 'error', 'failed', 'stopped')),
+ worker_id text COLLATE "C",
+ worker_generation bigint,
+ restart_attempts bigint NOT NULL DEFAULT 0,
+ next_restart_at bigint,
+ ever_live bigint NOT NULL DEFAULT 0,
+ last_error text COLLATE "C",
+ progress text COLLATE "C",
+ started_at bigint,
+ live_at bigint,
+ ended_at bigint,
+ created_at bigint NOT NULL,
+ updated_at bigint NOT NULL,
+ UNIQUE (session_id, destination_id)
+);
+
+CREATE TABLE output_logs (
+ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
+ output_id text COLLATE "C",
+ destination_id text COLLATE "C" NOT NULL,
+ level text COLLATE "C" NOT NULL CHECK (level IN ('info', 'warn', 'error')),
+ message text COLLATE "C" NOT NULL,
+ at bigint NOT NULL
+);
+
+CREATE TABLE recordings (
+ id text COLLATE "C" PRIMARY KEY,
+ session_id text COLLATE "C" NOT NULL UNIQUE REFERENCES ingest_sessions(id),
+ mode text COLLATE "C" NOT NULL CHECK (mode IN ('vod', 'clips')),
+ state text COLLATE "C" NOT NULL CHECK (state IN ('pending', 'requested', 'recording', 'finalizing', 'finalized', 'failed', 'cancelled')),
+ media_app text COLLATE "C",
+ media_vod_id text COLLATE "C",
+ attempts bigint NOT NULL DEFAULT 0,
+ next_attempt_at bigint NOT NULL DEFAULT 0,
+ last_error text COLLATE "C",
+ created_at bigint NOT NULL,
+ updated_at bigint NOT NULL
+);
+
+CREATE TABLE migration_map (
+ source_system text COLLATE "C" NOT NULL,
+ source_type text COLLATE "C" NOT NULL,
+ source_id text COLLATE "C" NOT NULL,
+ target_type text COLLATE "C",
+ target_id text COLLATE "C",
+ status text COLLATE "C" NOT NULL CHECK (status IN ('imported', 'held', 'excluded')),
+ reason text COLLATE "C",
+ imported_at bigint NOT NULL,
+ PRIMARY KEY (source_system, source_type, source_id)
+);
+
+CREATE TABLE leases (
+ name text COLLATE "C" PRIMARY KEY,
+ holder text COLLATE "C" NOT NULL,
+ expires_at bigint NOT NULL
+);
+
+CREATE INDEX idx_definitions_owner ON stream_definitions(owner_subject, state);
+CREATE INDEX idx_external_refs_ref ON external_refs(service, type, ref_id);
+CREATE INDEX idx_ingest_keys_definition ON ingest_keys(definition_id, status);
+CREATE INDEX idx_workers_kind_state ON workers(kind, state, generation);
+CREATE INDEX idx_sessions_definition ON ingest_sessions(definition_id, created_at);
+CREATE INDEX idx_sessions_state ON ingest_sessions(state);
+CREATE INDEX idx_sessions_worker ON ingest_sessions(worker_id, state);
+CREATE INDEX idx_transitions_session ON session_transitions(session_id, id);
+CREATE INDEX idx_destinations_definition ON destinations(definition_id);
+CREATE INDEX idx_outputs_worker ON outputs(worker_id, state);
+CREATE INDEX idx_outputs_session ON outputs(session_id);
+CREATE INDEX idx_output_logs_output ON output_logs(output_id, id);
+CREATE INDEX idx_output_logs_destination ON output_logs(destination_id, id);
+CREATE INDEX idx_recordings_state ON recordings(state, next_attempt_at);
+
+-- openvibe-sdk/events PostgreSQL outbox
+CREATE TABLE IF NOT EXISTS event_outbox (
+ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
+ event_id text NOT NULL UNIQUE,
+ envelope jsonb NOT NULL,
+ traceparent text,
+ created_at bigint NOT NULL,
+ attempts integer NOT NULL DEFAULT 0,
+ next_attempt_at bigint NOT NULL DEFAULT 0,
+ sent_at bigint,
+ seq bigint,
+ rejected_at bigint,
+ last_error text
+);
+CREATE INDEX IF NOT EXISTS event_outbox_due ON event_outbox (next_attempt_at, id) WHERE sent_at IS NULL AND rejected_at IS NULL;
+CREATE INDEX IF NOT EXISTS event_outbox_sent ON event_outbox (sent_at) WHERE sent_at IS NOT NULL;
@@ -13,15 +13,33 @@
@@ -13,15 +13,33 @@
"cookie-parser": "^1.4.7",
"cookie-parser": "^1.4.7",
"dotenv": "^16.4.7",
"dotenv": "^16.4.7",
"express": "^4.21.2",
"express": "^4.21.2",
+ "iovalkey": "^0.4.0",
"node-media-server": "2.7.4",
"node-media-server": "2.7.4",
- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.49.0",
- "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.12.0",
- "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.25.0"
+ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.76.0",
+ "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.21.3",
+ "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.28.0",
+ "pg": "^8.23.0"
+ },
+ "devDependencies": {
+ "@electric-sql/pglite": "^0.5.8"
},
},
"engines": {
"engines": {
"node": ">=22.12.0"
"node": ">=22.12.0"
}
}
},
},
+ "node_modules/@electric-sql/pglite": {
+ "version": "0.5.8",
+ "resolved": "https://registry.npmjs.org/@electric-sql/pglite/-/pglite-0.5.8.tgz",
+ "integrity": "sha512-n9tsbUOhwx2epK1V0ZG9Ar4SHWUju04dhmzZXiSBXwBoleOvIfals33NAaWgagQVAL4Rbvx/Ptsu3P+pA09f6Q==",
+ "devOptional": true,
+ "license": "Apache-2.0"
+ },
+ "node_modules/@iovalkey/commands": {
+ "version": "0.1.0",
+ "resolved": "https://registry.npmjs.org/@iovalkey/commands/-/commands-0.1.0.tgz",
+ "integrity": "sha512-/B9W4qKSSITDii5nkBCHyPkIkAi+ealUtr1oqBJsLxjSRLka4pxun2VvMNSmcwgAMxgXtQfl0qRv7TE+udPJzg==",
+ "license": "MIT"
+ },
"node_modules/accepts": {
"node_modules/accepts": {
"version": "1.3.8",
"version": "1.3.8",
"resolved": "https://registry.npmjs.org/accepts/-/accepts-1.3.8.tgz",
"resolved": "https://registry.npmjs.org/accepts/-/accepts-1.3.8.tgz",
@@ -263,6 +281,15 @@
@@ -263,6 +281,15 @@
"integrity": "sha512-jJ0bqzaylmJtVnNgzTeSOs8DPavpbYgEr/b0YL8/2GO3xJEhInFmhKMUnEJQjZumK7KXGFhUy89PrsJWlakBVg==",
"integrity": "sha512-jJ0bqzaylmJtVnNgzTeSOs8DPavpbYgEr/b0YL8/2GO3xJEhInFmhKMUnEJQjZumK7KXGFhUy89PrsJWlakBVg==",
"license": "ISC"
"license": "ISC"
},
},
+ "node_modules/cluster-key-slot": {
+ "version": "1.1.2",
+ "resolved": "https://registry.npmjs.org/cluster-key-slot/-/cluster-key-slot-1.1.2.tgz",
+ "integrity": "sha512-RMr0FhtfXemyinomL4hrWcYJxmX6deFdCxpJzhDttxgO1+bcCnkk+9drydLVDmAMG7NE6aN/fl4F7ucU/90gAA==",
+ "license": "Apache-2.0",
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
"node_modules/color-convert": {
"node_modules/color-convert": {
"version": "2.0.1",
"version": "2.0.1",
"resolved": "https://registry.npmjs.org/color-convert/-/color-convert-2.0.1.tgz",
"resolved": "https://registry.npmjs.org/color-convert/-/color-convert-2.0.1.tgz",
@@ -372,6 +399,15 @@
@@ -372,6 +399,15 @@
"node": ">=4.0.0"
"node": ">=4.0.0"
}
}
},
},
+ "node_modules/denque": {
+ "version": "2.1.0",
+ "resolved": "https://registry.npmjs.org/denque/-/denque-2.1.0.tgz",
+ "integrity": "sha512-HVQE3AAb/pxF8fQAoiqpvg9i3evqug3hoiwakOyZAwJm+6vZehbkYXZ0l4JxS+I3QxM97v5aaRNhj8v5oBhekw==",
+ "license": "Apache-2.0",
+ "engines": {
+ "node": ">=0.10"
+ }
+ },
"node_modules/depd": {
"node_modules/depd": {
"version": "2.0.0",
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/depd/-/depd-2.0.0.tgz",
"resolved": "https://registry.npmjs.org/depd/-/depd-2.0.0.tgz",
@@ -802,6 +838,49 @@
@@ -802,6 +838,49 @@
"integrity": "sha512-JV/yugV2uzW5iMRSiZAyDtQd+nxtUnjeLt0acNdw98kKLrvuRVyB80tsREOE7yvGVgalhZ6RNXCmEHkUKBKxew==",
"integrity": "sha512-JV/yugV2uzW5iMRSiZAyDtQd+nxtUnjeLt0acNdw98kKLrvuRVyB80tsREOE7yvGVgalhZ6RNXCmEHkUKBKxew==",
"license": "ISC"
"license": "ISC"
},
},
+ "node_modules/iovalkey": {
+ "version": "0.4.0",
+ "resolved": "https://registry.npmjs.org/iovalkey/-/iovalkey-0.4.0.tgz",
+ "integrity": "sha512-OSUKxJ+s44CLdUTaicX4+pVrBN/zHKIYwr2oKhRDuczr19FwcSCXEZHzcMUYWZNh1mNSCV9bNJHW/T6cHVm9Aw==",
+ "license": "MIT",
+ "dependencies": {
+ "@iovalkey/commands": "^0.1.0",
+ "cluster-key-slot": "^1.1.0",
+ "debug": "^4.3.4",
+ "denque": "^2.1.0",
+ "lodash.defaults": "^4.2.0",
+ "lodash.isarguments": "^3.1.0",
+ "redis-errors": "^1.2.0",
+ "redis-parser": "^3.0.0",
+ "standard-as-callback": "^2.1.0"
+ },
+ "engines": {
+ "node": ">=18.12.0"
+ }
+ },
+ "node_modules/iovalkey/node_modules/debug": {
+ "version": "4.4.3",
+ "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz",
+ "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==",
+ "license": "MIT",
+ "dependencies": {
+ "ms": "^2.1.3"
+ },
+ "engines": {
+ "node": ">=6.0"
+ },
+ "peerDependenciesMeta": {
+ "supports-color": {
+ "optional": true
+ }
+ }
+ },
+ "node_modules/iovalkey/node_modules/ms": {
+ "version": "2.1.3",
+ "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz",
+ "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==",
+ "license": "MIT"
+ },
"node_modules/ipaddr.js": {
"node_modules/ipaddr.js": {
"version": "1.9.1",
"version": "1.9.1",
"resolved": "https://registry.npmjs.org/ipaddr.js/-/ipaddr.js-1.9.1.tgz",
"resolved": "https://registry.npmjs.org/ipaddr.js/-/ipaddr.js-1.9.1.tgz",
@@ -872,12 +951,24 @@
@@ -872,12 +951,24 @@
"integrity": "sha512-dMInicTPVE8d1e5otfwmmjlxkZoUpiVLwyeTdUsi/Caj/gfzzblBcCE5sRHV/AsjuCmxWrte2TNGSYuCeCq+0Q==",
"integrity": "sha512-dMInicTPVE8d1e5otfwmmjlxkZoUpiVLwyeTdUsi/Caj/gfzzblBcCE5sRHV/AsjuCmxWrte2TNGSYuCeCq+0Q==",
"license": "MIT"
"license": "MIT"
},
},
+ "node_modules/lodash.defaults": {
+ "version": "4.2.0",
+ "resolved": "https://registry.npmjs.org/lodash.defaults/-/lodash.defaults-4.2.0.tgz",
+ "integrity": "sha512-qjxPLHd3r5DnsdGacqOMU6pb/avJzdh9tFX2ymgoZE27BmjXrNy/y4LoaiTeAb+O3gL8AfpJGtqfX/ae2leYYQ==",
+ "license": "MIT"
+ },
"node_modules/lodash.includes": {
"node_modules/lodash.includes": {
"version": "4.3.0",
"version": "4.3.0",
"resolved": "https://registry.npmjs.org/lodash.includes/-/lodash.includes-4.3.0.tgz",
"resolved": "https://registry.npmjs.org/lodash.includes/-/lodash.includes-4.3.0.tgz",
"integrity": "sha512-W3Bx6mdkRTGtlJISOvVD/lbqjTlPPUDTMnlXZFnVwi9NKJ6tiAk6LVdlhZMm17VZisqhKcgzpO5Wz91PCt5b0w==",
"integrity": "sha512-W3Bx6mdkRTGtlJISOvVD/lbqjTlPPUDTMnlXZFnVwi9NKJ6tiAk6LVdlhZMm17VZisqhKcgzpO5Wz91PCt5b0w==",
"license": "MIT"
"license": "MIT"
},
},
+ "node_modules/lodash.isarguments": {
+ "version": "3.1.0",
+ "resolved": "https://registry.npmjs.org/lodash.isarguments/-/lodash.isarguments-3.1.0.tgz",
+ "integrity": "sha512-chi4NHZlZqZD18a0imDHnZPrDeBbTtVN7GXMwuGdRH9qotxAjYs3aVLKc7zNOG9eddR5Ksd8rvFEBc9SsggPpg==",
+ "license": "MIT"
+ },
"node_modules/lodash.isboolean": {
"node_modules/lodash.isboolean": {
"version": "3.0.3",
"version": "3.0.3",
"resolved": "https://registry.npmjs.org/lodash.isboolean/-/lodash.isboolean-3.0.3.tgz",
"resolved": "https://registry.npmjs.org/lodash.isboolean/-/lodash.isboolean-3.0.3.tgz",
@@ -1115,9 +1206,9 @@
@@ -1115,9 +1206,9 @@
}
}
},
},
"node_modules/openvibe-contracts": {
"node_modules/openvibe-contracts": {
- "version": "0.49.0",
- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.49.0",
- "integrity": "sha512-lapsLoBm3ZpndgiOMELdCBh2PJtuuyqj6SUhQsfwJkxnLdQfyhXfRFgiuLh+iQrg4kqtG9611191Mp8zAfk7fQ==",
+ "version": "0.76.0",
+ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.76.0",
+ "integrity": "sha512-zTdx776KWrcJKm1JTnf+0+BnsvswdvXy1ldPyse1hSqhnW/SSr98sX696eRetFUhTl8qVM1xSRCykqIUbqItPw==",
"license": "AGPL-3.0-only",
"license": "AGPL-3.0-only",
"dependencies": {
"dependencies": {
"ajv": "^8.17.1",
"ajv": "^8.17.1",
@@ -1131,18 +1222,38 @@
@@ -1131,18 +1222,38 @@
}
}
},
},
"node_modules/openvibe-sdk": {
"node_modules/openvibe-sdk": {
- "version": "0.12.0",
- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.12.0",
- "integrity": "sha512-iG1h905O1f7y/PnP8xLA6GCVzXEBDfSaBj2tc1ucNreWg2sT/AJWosb8NW0WRMK5WS4sIQ6tMsprmDLj/fjjeQ==",
+ "version": "0.21.3",
+ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.21.3",
+ "integrity": "sha512-oAcxznSHKko+7zQCHS2VrSW2gqpXgbagnu+UZ60vrW/QjwtoCvatMZvtgEwpekYFGzpI7xklvCe0EOb1fa1H2g==",
"license": "MIT",
"license": "MIT",
"engines": {
"engines": {
"node": ">=20"
"node": ">=20"
+ },
+ "peerDependencies": {
+ "@electric-sql/pglite": "^0.5.8",
+ "better-sqlite3": ">=11",
+ "iovalkey": "^0.4.0",
+ "pg": "^8.23.0"
+ },
+ "peerDependenciesMeta": {
+ "@electric-sql/pglite": {
+ "optional": true
+ },
+ "better-sqlite3": {
+ "optional": true
+ },
+ "iovalkey": {
+ "optional": true
+ },
+ "pg": {
+ "optional": true
+ }
}
}
},
},
"node_modules/openvibe-shared": {
"node_modules/openvibe-shared": {
- "version": "1.25.0",
- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.25.0",
- "integrity": "sha512-oqZ+PvljrWi8IEbtg6bDYai17ZV2sALFx/ezMptniELSr10TgFUiGUQt+b1IWwiHv0T2Y1U2iSrkirJxMNgjSw==",
+ "version": "1.28.0",
+ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.28.0",
+ "integrity": "sha512-WLd11lDG75veVi7mFwngquK2nqIQV86Z14Ad/xE2dYvyMT5mhKJxxxlKUoskpdyLeap0ucMvzzS8Tg7vPGwYoQ==",
"license": "MIT",
"license": "MIT",
"dependencies": {
"dependencies": {
"jsonwebtoken": "^9.0.0"
"jsonwebtoken": "^9.0.0"
@@ -1163,6 +1274,134 @@
@@ -1163,6 +1274,134 @@
"integrity": "sha512-A/AGNMFN3c8bOlvV9RreMdrv7jsmF9XIfDeCd87+I8RNg6s78BhJxMu69NEMHBSJFxKidViTEdruRwEk/WIKqA==",
"integrity": "sha512-A/AGNMFN3c8bOlvV9RreMdrv7jsmF9XIfDeCd87+I8RNg6s78BhJxMu69NEMHBSJFxKidViTEdruRwEk/WIKqA==",
"license": "MIT"
"license": "MIT"
},
},
+ "node_modules/pg": {
+ "version": "8.23.0",
+ "resolved": "https://registry.npmjs.org/pg/-/pg-8.23.0.tgz",
+ "integrity": "sha512-Ip2EQCngowJLGOfCwkFhPXU7/ljlhn6Rxlmy4XYfL2Y+vyRM59+8uR2xqRWKdYmbXmxCFOAmKxBuSUCdF34qLg==",
+ "license": "MIT",
+ "dependencies": {
+ "pg-connection-string": "^2.14.0",
+ "pg-pool": "^3.14.0",
+ "pg-protocol": "^1.16.0",
+ "pg-types": "2.2.0",
+ "pgpass": "1.0.5"
+ },
+ "engines": {
+ "node": ">= 16.0.0"
+ },
+ "optionalDependencies": {
+ "pg-cloudflare": "^1.4.0"
+ },
+ "peerDependencies": {
+ "pg-native": ">=3.0.1"
+ },
+ "peerDependenciesMeta": {
+ "pg-native": {
+ "optional": true
+ }
+ }
+ },
+ "node_modules/pg-cloudflare": {
+ "version": "1.4.0",
+ "resolved": "https://registry.npmjs.org/pg-cloudflare/-/pg-cloudflare-1.4.0.tgz",
+ "integrity": "sha512-Vo7z/6rrQYxpNRylp4Tlob2elzbh+N/MOQbxFVWCxS7oEx6jF53GTJFxK2WWpKuBRkmiin4Mt+xofFDjx09R0A==",
+ "license": "MIT",
+ "optional": true
+ },
+ "node_modules/pg-connection-string": {
+ "version": "2.14.0",
+ "resolved": "https://registry.npmjs.org/pg-connection-string/-/pg-connection-string-2.14.0.tgz",
+ "integrity": "sha512-XwWDGcLRGCXAR8F/AM5bG7Q+A3Wm2s6QeEjlOKZLlH3UYcguiqCWKyWXVag5TLTIjR7oOJUY8kcADaZgWPyLeg==",
+ "license": "MIT"
+ },
+ "node_modules/pg-int8": {
+ "version": "1.0.1",
+ "resolved": "https://registry.npmjs.org/pg-int8/-/pg-int8-1.0.1.tgz",
+ "integrity": "sha512-WCtabS6t3c8SkpDBUlb1kjOs7l66xsGdKpIPZsg4wR+B3+u9UAum2odSsF9tnvxg80h4ZxLWMy4pRjOsFIqQpw==",
+ "license": "ISC",
+ "engines": {
+ "node": ">=4.0.0"
+ }
+ },
+ "node_modules/pg-pool": {
+ "version": "3.14.0",
+ "resolved": "https://registry.npmjs.org/pg-pool/-/pg-pool-3.14.0.tgz",
+ "integrity": "sha512-gKtPkFdQPU3DksooVLi9LsjZxrsBUZIpa+7aVx+LV5pNh0KzP4Zleud2po+ConrxbuXGBJ6Hfer6hdgpIBpBaw==",
+ "license": "MIT",
+ "peerDependencies": {
+ "pg": ">=8.0"
+ }
+ },
+ "node_modules/pg-protocol": {
+ "version": "1.16.0",
+ "resolved": "https://registry.npmjs.org/pg-protocol/-/pg-protocol-1.16.0.tgz",
+ "integrity": "sha512-sILXutLVjCLjcDuOmvhX5e2Z4cS5qG/6Bu3VkpFwdf/633ElGLpEh9bgmuI5I4sqKqkifQiGyiCcx1HdtrK7tg==",
+ "license": "MIT"
+ },
+ "node_modules/pg-types": {
+ "version": "2.2.0",
+ "resolved": "https://registry.npmjs.org/pg-types/-/pg-types-2.2.0.tgz",
+ "integrity": "sha512-qTAAlrEsl8s4OiEQY69wDvcMIdQN6wdz5ojQiOy6YRMuynxenON0O5oCpJI6lshc6scgAY8qvJ2On/p+CXY0GA==",
+ "license": "MIT",
+ "dependencies": {
+ "pg-int8": "1.0.1",
+ "postgres-array": "~2.0.0",
+ "postgres-bytea": "~1.0.0",
+ "postgres-date": "~1.0.4",
+ "postgres-interval": "^1.1.0"
+ },
+ "engines": {
+ "node": ">=4"
+ }
+ },
+ "node_modules/pgpass": {
+ "version": "1.0.5",
+ "resolved": "https://registry.npmjs.org/pgpass/-/pgpass-1.0.5.tgz",
+ "integrity": "sha512-FdW9r/jQZhSeohs1Z3sI1yxFQNFvMcnmfuj4WBMUTxOrAyLMaTcE1aAMBiTlbMNaXvBCQuVi0R7hd8udDSP7ug==",
+ "license": "MIT",
+ "dependencies": {
+ "split2": "^4.1.0"
+ }
+ },
+ "node_modules/postgres-array": {
+ "version": "2.0.0",
+ "resolved": "https://registry.npmjs.org/postgres-array/-/postgres-array-2.0.0.tgz",
+ "integrity": "sha512-VpZrUqU5A69eQyW2c5CA1jtLecCsN2U/bD6VilrFDWq5+5UIEVO7nazS3TEcHf1zuPYO/sqGvUvW62g86RXZuA==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=4"
+ }
+ },
+ "node_modules/postgres-bytea": {
+ "version": "1.0.1",
+ "resolved": "https://registry.npmjs.org/postgres-bytea/-/postgres-bytea-1.0.1.tgz",
+ "integrity": "sha512-5+5HqXnsZPE65IJZSMkZtURARZelel2oXUEO8rH83VS/hxH5vv1uHquPg5wZs8yMAfdv971IU+kcPUczi7NVBQ==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
+ "node_modules/postgres-date": {
+ "version": "1.0.7",
+ "resolved": "https://registry.npmjs.org/postgres-date/-/postgres-date-1.0.7.tgz",
+ "integrity": "sha512-suDmjLVQg78nMK2UZ454hAG+OAW+HQPZ6n++TNDUX+L0+uUlLywnoxJKDou51Zm+zTCjrCl0Nq6J9C5hP9vK/Q==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
+ "node_modules/postgres-interval": {
+ "version": "1.2.0",
+ "resolved": "https://registry.npmjs.org/postgres-interval/-/postgres-interval-1.2.0.tgz",
+ "integrity": "sha512-9ZhXKM/rw350N1ovuWHbGxnGh/SNJ4cnxHiM0rxE4VN41wsg8P8zWn9hv/buK00RP4WvlOyr/RBDiptyxVbkZQ==",
+ "license": "MIT",
+ "dependencies": {
+ "xtend": "^4.0.0"
+ },
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
"node_modules/prebuild-install": {
"node_modules/prebuild-install": {
"version": "7.1.3",
"version": "7.1.3",
"resolved": "https://registry.npmjs.org/prebuild-install/-/prebuild-install-7.1.3.tgz",
"resolved": "https://registry.npmjs.org/prebuild-install/-/prebuild-install-7.1.3.tgz",
@@ -1286,6 +1525,27 @@
@@ -1286,6 +1525,27 @@
"node": ">= 6"
"node": ">= 6"
}
}
},
},
+ "node_modules/redis-errors": {
+ "version": "1.2.0",
+ "resolved": "https://registry.npmjs.org/redis-errors/-/redis-errors-1.2.0.tgz",
+ "integrity": "sha512-1qny3OExCf0UvUV/5wpYKf2YwPcOqXzkwKKSmKHiE6ZMQs5heeE/c8eXK+PNllPvmjgAbfnsbpkGZWy8cBpn9w==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=4"
+ }
+ },
+ "node_modules/redis-parser": {
+ "version": "3.0.0",
+ "resolved": "https://registry.npmjs.org/redis-parser/-/redis-parser-3.0.0.tgz",
+ "integrity": "sha512-DJnGAeenTdpMEH6uAJRK/uiyEIH9WVsUmoLwzudwGJUwZPp80PDBWPHXSAGNPwNvIXAbe7MSUB1zQFugFml66A==",
+ "license": "MIT",
+ "dependencies": {
+ "redis-errors": "^1.0.0"
+ },
+ "engines": {
+ "node": ">=4"
+ }
+ },
"node_modules/require-from-string": {
"node_modules/require-from-string": {
"version": "2.0.2",
"version": "2.0.2",
"resolved": "https://registry.npmjs.org/require-from-string/-/require-from-string-2.0.2.tgz",
"resolved": "https://registry.npmjs.org/require-from-string/-/require-from-string-2.0.2.tgz",
@@ -1501,6 +1761,21 @@
@@ -1501,6 +1761,21 @@
"simple-concat": "^1.0.0"
"simple-concat": "^1.0.0"
}
}
},
},
+ "node_modules/split2": {
+ "version": "4.2.0",
+ "resolved": "https://registry.npmjs.org/split2/-/split2-4.2.0.tgz",
+ "integrity": "sha512-UcjcJOWknrNkF6PLX83qcHM6KHgVKNkV62Y8a5uYDVv9ydGQVwAHMKqHdJje1VTWpljG0WYpCDhrCdAOYH4TWg==",
+ "license": "ISC",
+ "engines": {
+ "node": ">= 10.x"
+ }
+ },
+ "node_modules/standard-as-callback": {
+ "version": "2.1.0",
+ "resolved": "https://registry.npmjs.org/standard-as-callback/-/standard-as-callback-2.1.0.tgz",
+ "integrity": "sha512-qoRRSyROncaz1z0mvYqIE4lCd9p2R90i6GxW3uZv5ucSu8tU7B5HXUP1gG8pVZsYNVaXjk8ClXHPttLyxAL48A==",
+ "license": "MIT"
+ },
"node_modules/statuses": {
"node_modules/statuses": {
"version": "2.0.2",
"version": "2.0.2",
"resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.2.tgz",
"resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.2.tgz",
@@ -1670,6 +1945,15 @@
@@ -1670,6 +1945,15 @@
"optional": true
"optional": true
}
}
}
}
+ },
+ "node_modules/xtend": {
+ "version": "4.0.2",
+ "resolved": "https://registry.npmjs.org/xtend/-/xtend-4.0.2.tgz",
+ "integrity": "sha512-LKYU1iAXJXUgAXn9URjiu+MWhyUXHsvfp7mcuYm9dSUKK0/CjtrUwFAxD82/mCWbtLsGjFIad0wIsod4zrTAEQ==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=0.4"
+ }
}
}
}
}
}
}
@@ -11,9 +11,16 @@
@@ -11,9 +11,16 @@
"rtmp-ingest": "node workers/rtmp-ingest.js",
"rtmp-ingest": "node workers/rtmp-ingest.js",
"restream-worker": "node workers/restream-worker.js",
"restream-worker": "node workers/restream-worker.js",
"migrate-from-live": "node scripts/migrate-from-live.js",
"migrate-from-live": "node scripts/migrate-from-live.js",
- "test": "node test/run.js"
+ "test": "node test/run.js",
+ "test:pg": "OPENRE_TEST_STORE=pg node test/run.js"
},
},
- "keywords": ["openvibe", "openre", "rtmp", "restream", "ingest"],
+ "keywords": [
+ "openvibe",
+ "openre",
+ "rtmp",
+ "restream",
+ "ingest"
+ ],
"author": "OpenVibers",
"author": "OpenVibers",
"license": "AGPL-3.0-only",
"license": "AGPL-3.0-only",
"repository": "github:OpenVibers/OpenRe.Stream",
"repository": "github:OpenVibers/OpenRe.Stream",
@@ -22,12 +29,17 @@
@@ -22,12 +29,17 @@
"cookie-parser": "^1.4.7",
"cookie-parser": "^1.4.7",
"dotenv": "^16.4.7",
"dotenv": "^16.4.7",
"express": "^4.21.2",
"express": "^4.21.2",
+ "iovalkey": "^0.4.0",
"node-media-server": "2.7.4",
"node-media-server": "2.7.4",
- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.49.0",
- "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.12.0",
- "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.25.0"
+ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.76.0",
+ "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.21.3",
+ "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.28.0",
+ "pg": "^8.23.0"
},
},
"engines": {
"engines": {
"node": ">=22.12.0"
"node": ">=22.12.0"
+ },
+ "devDependencies": {
+ "@electric-sql/pglite": "^0.5.8"
}
}
}
}
@@ -15,7 +15,8 @@
@@ -15,7 +15,8 @@
* bind OPENRE_RTMP_BIND is public and the RTMP port really listens on a public address
* bind OPENRE_RTMP_BIND is public and the RTMP port really listens on a public address
* dns the ingest host resolves (A/AAAA) only to this host's addresses
* dns the ingest host resolves (A/AAAA) only to this host's addresses
* port the ingest port answers an RTMP handshake from outside (see "port" below)
* port the ingest port answers an RTMP handshake from outside (see "port" below)
- * db PRAGMA integrity_check and foreign_key_check of openre.db (opened read-only)
+ * db OpenRe's PostgreSQL database (DATABASE_URL of openre.env, one READ ONLY transaction): no data
+ * checksum failure, no foreign key left unvalidated, the main tables' counts
* live-env /etc/openvibe/live.env has OPENRE_URL, OPENRE_EVENTS_SECRET (32+ chars) and
* live-env /etc/openvibe/live.env has OPENRE_URL, OPENRE_EVENTS_SECRET (32+ chars) and
* OV_OAUTH_CLIENT_SECRET, and the running Live process was started with them
* OV_OAUTH_CLIENT_SECRET, and the running Live process was started with them
* events Events has Live's openre.session.* subscription to /internal/openre-events, enabled,
* events Events has Live's openre.session.* subscription to /internal/openre-events, enabled,
@@ -27,7 +28,10 @@
@@ -27,7 +28,10 @@
* that is not the host (a checkout with `npm ci` is enough), or pass --probe-url with a probe you
* that is not the host (a checkout with `npm ci` is enough), or pass --probe-url with a probe you
* trust (GET, `{host}` and `{port}` replaced, JSON answer with a boolean `open` or `reachable`).
* trust (GET, `{host}` and `{port}` replaced, JSON answer with a boolean `open` or `reachable`).
*
*
- * Nothing is written anywhere: files and databases are opened read-only, no secret value is
+ * OpenRe and Events serve from PostgreSQL (ADR-035): their databases are read through the DATABASE_URL of their env
+ * files (--openre-db / --events-db take env:<file>, a postgres:// URL or a SQLite file); Live's is still its SQLite file.
+ *
+ * Nothing is written anywhere: files are opened read-only and PostgreSQL is read in READ ONLY transactions, no secret value is
* printed, the only network traffic is DNS, one GET to /api/ready and one TCP connection with an
* printed, the only network traffic is DNS, one GET to /api/ready and one TCP connection with an
* RTMP handshake (no publish). Exit 0 when no check FAILs (--strict: also no WARN/MANUAL).
* RTMP handshake (no publish). Exit 0 when no check FAILs (--strict: also no WARN/MANUAL).
*/
*/
@@ -43,8 +47,8 @@ const DEFAULTS = Object.freeze({
@@ -43,8 +47,8 @@ const DEFAULTS = Object.freeze({
root: '/opt/openre.stream',
root: '/opt/openre.stream',
openreEnv: '/etc/openvibe/openre.env',
openreEnv: '/etc/openvibe/openre.env',
liveEnv: '/etc/openvibe/live.env',
liveEnv: '/etc/openvibe/live.env',
- openreDb: '/var/lib/openre/openre.db',
- eventsDb: '/var/lib/openvibe-events/events.db',
+ openreDb: 'env:/etc/openvibe/openre.env',
+ eventsDb: 'env:/etc/openvibe/events.env',
liveDb: '/opt/openvibe.live/data/live.db',
liveDb: '/opt/openvibe.live/data/live.db',
api: 'http://127.0.0.1:4500',
api: 'http://127.0.0.1:4500',
openreUrl: 'http://127.0.0.1:4500',
openreUrl: 'http://127.0.0.1:4500',
@@ -134,17 +138,44 @@ function realDeps() {
@@ -134,17 +138,44 @@ function realDeps() {
interfaces: () => os.networkInterfaces(),
interfaces: () => os.networkInterfaces(),
probe: rtmpProbe,
probe: rtmpProbe,
fetch: (url, opts) => fetch(url, opts),
fetch: (url, opts) => fetch(url, opts),
- openDb: (p) => {
+ openDb: (target) => {
+ if (/^postgres(ql)?:\/\//.test(target)) {
+ const silent = { log() {}, info() {}, warn() {}, error() {} };
+ return require('openvibe-sdk/db').createDb({ url: target, service: 'openre-preflight', max: 1, log: silent });
+ }
const Database = require('better-sqlite3');
const Database = require('better-sqlite3');
- return new Database(p, { readonly: true, fileMustExist: true });
+ return new Database(target, { readonly: true, fileMustExist: true });
},
},
};
};
}
}
-function withDb(deps, file, fn) {
+/**
+ * fn(db, kind) on a read-only handle. `target`: env:<file> (that file's DATABASE_URL, never printed), a postgres:// URL
+ * or a SQLite file. PostgreSQL is read in one READ ONLY transaction (a write fails); SQLite is opened read-only.
+ */
+async function withDb(deps, target, fn) {
+ let where = String(target);
let db;
let db;
- try { db = deps.openDb(file); } catch (err) { return { error: `cannot open ${file} read-only: ${err.message}` }; }
- try { return fn(db); } finally { try { db.close(); } catch { /* read-only */ } }
+ try {
+ let url = where;
+ if (where.startsWith('env:')) {
+ const file = where.slice(4);
+ where = `the DATABASE_URL of ${file}`;
+ const env = parseEnvFile(deps.readFile(file));
+ if (!env) return { error: `cannot read ${file}` };
+ if (!env.DATABASE_URL) return { error: `${file} has no DATABASE_URL` };
+ url = env.DATABASE_URL;
+ } else if (/^postgres(ql)?:\/\//.test(where)) where = 'the PostgreSQL URL given';
+ db = await deps.openDb(url);
+ } catch (err) { return { error: `cannot open ${where} read-only: ${err.message}` }; }
+ try {
+ if (typeof db.tx === 'function') {
+ return await db.tx(async () => { await db.exec('SET TRANSACTION READ ONLY'); return await fn(db, 'postgresql'); });
+ }
+ return await fn(db, 'sqlite');
+ } catch (err) {
+ return { error: `${where}: ${err.message}` };
+ } finally { try { await db.close(); } catch { /* read-only */ } }
}
}
// ── checks ──────────────────────────────────────────────────────────────────
// ── checks ──────────────────────────────────────────────────────────────────
@@ -191,25 +222,25 @@ async function checkService(ctx) {
@@ -191,25 +222,25 @@ async function checkService(ctx) {
if (res.status !== 200 || body.status !== 'ready') problems.push(`status ${res.status} ${body.status} (${JSON.stringify(body.checks || {})})`);
if (res.status !== 200 || body.status !== 'ready') problems.push(`status ${res.status} ${body.status} (${JSON.stringify(body.checks || {})})`);
if (body.mode === 'drill') problems.push('the API runs in drill mode (OPENRE_DRILL)');
if (body.mode === 'drill') problems.push('the API runs in drill mode (OPENRE_DRILL)');
const workers = Array.isArray(body.workers) ? body.workers : [];
const workers = Array.isArray(body.workers) ? body.workers : [];
- const rows = ctx.workerRows || workerRowsFromDb(ctx);
+ const rows = ctx.workerRows || await workerRowsFromDb(ctx);
for (const kind of WORKER_KINDS) {
for (const kind of WORKER_KINDS) {
- const ready = workers.filter((w) => w.kind === kind && w.state === 'ready');
+ const ready = await workers.filter((w) => w.kind === kind && w.state === 'ready');
if (ready.length !== 1) { problems.push(`${ready.length} ready ${kind} workers (want 1)`); continue; }
if (ready.length !== 1) { problems.push(`${ready.length} ready ${kind} workers (want 1)`); continue; }
const row = rows && rows.find((w) => w.kind === kind && w.generation === ready[0].generation);
const row = rows && rows.find((w) => w.kind === kind && w.generation === ready[0].generation);
if (ctx.release && row && row.release && row.release !== ctx.release) problems.push(`${kind}#${ready[0].generation} runs release ${row.release}, current is ${ctx.release}: deploy.sh workers`);
if (ctx.release && row && row.release && row.release !== ctx.release) problems.push(`${kind}#${ready[0].generation} runs release ${row.release}, current is ${ctx.release}: deploy.sh workers`);
}
}
- const draining = workers.filter((w) => w.state === 'draining').map((w) => `${w.kind}#${w.generation}`);
+ const draining = (await workers.filter((w) => w.state === 'draining')).map((w) => `${w.kind}#${w.generation}`);
if (!body.coordinator || !body.coordinator.lease_valid) problems.push('no valid coordinator lease');
if (!body.coordinator || !body.coordinator.lease_valid) problems.push('no valid coordinator lease');
if (!body.events || !body.events.configured) problems.push('event relay not configured (EVENTS_URL, OV_OAUTH_CLIENT_SECRET)');
if (!body.events || !body.events.configured) problems.push('event relay not configured (EVENTS_URL, OV_OAUTH_CLIENT_SECRET)');
if (problems.length) return ['FAIL', problems.join('; ')];
if (problems.length) return ['FAIL', problems.join('; ')];
- const summary = WORKER_KINDS.map((k) => { const w = workers.find((x) => x.kind === k && x.state === 'ready'); return `${k}#${w.generation}`; }).join(', ');
+ const summary = (await Promise.all(WORKER_KINDS.map(async (k) => { const w = await workers.find((x) => x.kind === k && x.state === 'ready'); return `${k}#${w.generation}`; }))).join(', ');
const pending = body.events && body.events.pending ? `, ${body.events.pending} events waiting in the outbox` : '';
const pending = body.events && body.events.pending ? `, ${body.events.pending} events waiting in the outbox` : '';
if (draining.length) return ['WARN', `ready: ${summary}; still draining: ${draining.join(', ')}${pending}`];
if (draining.length) return ['WARN', `ready: ${summary}; still draining: ${draining.join(', ')}${pending}`];
return ['PASS', `ready: ${summary}, coordinator lease valid, event relay on${pending}`];
return ['PASS', `ready: ${summary}, coordinator lease valid, event relay on${pending}`];
}
}
-function workerRowsFromDb(ctx) {
- const r = withDb(ctx.deps, ctx.opts.openreDb, (db) => db.prepare("SELECT kind, generation, release, state FROM workers WHERE state IN ('starting', 'ready', 'draining')").all());
+async function workerRowsFromDb(ctx) {
+ const r = await withDb(ctx.deps, ctx.opts.openreDb, async (db) => await db.prepare("SELECT kind, generation, release, state FROM workers WHERE state IN ('starting', 'ready', 'draining')").all());
ctx.workerRows = Array.isArray(r) ? r : null;
ctx.workerRows = Array.isArray(r) ? r : null;
return ctx.workerRows;
return ctx.workerRows;
}
}
@@ -312,20 +343,30 @@ async function checkPort(ctx) {
@@ -312,20 +343,30 @@ async function checkPort(ctx) {
}
}
async function checkDb(ctx) {
async function checkDb(ctx) {
- const r = withDb(ctx.deps, ctx.opts.openreDb, (db) => {
- const integrity = db.prepare('PRAGMA integrity_check').all().map((x) => Object.values(x)[0]).join('; ');
- const fk = db.prepare('PRAGMA foreign_key_check').all();
+ const r = await withDb(ctx.deps, ctx.opts.openreDb, async (db, kind) => {
+ let integrity = 'ok';
+ let fk = 0;
+ if (kind === 'sqlite') {
+ integrity = (await db.prepare('PRAGMA integrity_check').all()).map((x) => Object.values(x)[0]).join('; ');
+ fk = (await db.prepare('PRAGMA foreign_key_check').all()).length;
+ } else {
+ // PostgreSQL checks every key on write: what can still be wrong is a data page (checksums) or a key added NOT VALID.
+ const c = await db.prepare('SELECT checksum_failures AS n FROM pg_stat_database WHERE datname = current_database()').get();
+ if (c && Number(c.n) > 0) integrity = `${c.n} data checksum failure(s)`;
+ fk = Number((await db.prepare("SELECT count(*) AS n FROM pg_constraint WHERE contype = 'f' AND NOT convalidated").get()).n);
+ }
const counts = {};
const counts = {};
for (const t of ['stream_definitions', 'ingest_keys', 'ingest_sessions', 'destinations', 'migration_map']) {
for (const t of ['stream_definitions', 'ingest_keys', 'ingest_sessions', 'destinations', 'migration_map']) {
- try { counts[t] = db.prepare(`SELECT count(*) AS n FROM ${t}`).get().n; } catch { counts[t] = null; }
+ try { counts[t] = (await db.prepare(`SELECT count(*) AS n FROM ${t}`).get()).n; } catch { counts[t] = null; }
}
}
- return { integrity, fk: fk.length, counts };
+ return { integrity, fk, kind, counts };
});
});
if (r.error) return ['FAIL', r.error];
if (r.error) return ['FAIL', r.error];
const counts = Object.entries(r.counts).map(([k, v]) => `${k} ${v == null ? '?' : v}`).join(', ');
const counts = Object.entries(r.counts).map(([k, v]) => `${k} ${v == null ? '?' : v}`).join(', ');
- if (r.integrity !== 'ok') return ['FAIL', `integrity_check = ${r.integrity.slice(0, 300)}`];
- if (r.fk) return ['FAIL', `integrity_check ok, but ${r.fk} foreign key violation(s)`];
- return ['PASS', `integrity_check ok, no foreign key violations (${counts})`];
+ const what = r.kind === 'sqlite' ? 'integrity_check' : 'PostgreSQL data checksums';
+ if (r.integrity !== 'ok') return ['FAIL', `${what} = ${r.integrity.slice(0, 300)}`];
+ if (r.fk) return ['FAIL', `${what} ok, but ${r.fk} foreign key ${r.kind === 'sqlite' ? 'violation(s)' : 'constraint(s) not validated'}`];
+ return ['PASS', `${what} ok, no foreign key ${r.kind === 'sqlite' ? 'violations' : 'left unvalidated'} (${counts})`];
}
}
async function checkLiveEnv(ctx) {
async function checkLiveEnv(ctx) {
@@ -360,10 +401,10 @@ async function checkLiveEnv(ctx) {
@@ -360,10 +401,10 @@ async function checkLiveEnv(ctx) {
async function checkEvents(ctx) {
async function checkEvents(ctx) {
const { opts } = ctx;
const { opts } = ctx;
const env = liveEnv(ctx) || {};
const env = liveEnv(ctx) || {};
- const r = withDb(ctx.deps, opts.eventsDb, (db) => {
- const subs = db.prepare("SELECT id, enabled, secret FROM subscriptions WHERE consumer = 'live' AND topic_pattern = ? AND endpoint = ?").all(opts.topic, opts.endpoint);
- const dead = subs.length ? db.prepare("SELECT count(*) AS n FROM deliveries WHERE subscription_id = ? AND status = 'dead'").get(subs[0].id).n : 0;
- const pending = subs.length ? db.prepare("SELECT count(*) AS n FROM deliveries WHERE subscription_id = ? AND status IN ('pending', 'failed')").get(subs[0].id).n : 0;
+ const r = await withDb(ctx.deps, opts.eventsDb, async (db) => {
+ const subs = await db.prepare("SELECT id, enabled, secret FROM subscriptions WHERE consumer = 'live' AND topic_pattern = ? AND endpoint = ?").all(opts.topic, opts.endpoint);
+ const dead = subs.length ? (await db.prepare("SELECT count(*) AS n FROM deliveries WHERE subscription_id = ? AND status = 'dead'").get(subs[0].id)).n : 0;
+ const pending = subs.length ? (await db.prepare("SELECT count(*) AS n FROM deliveries WHERE subscription_id = ? AND status IN ('pending', 'failed')").get(subs[0].id)).n : 0;
return { subs, dead, pending };
return { subs, dead, pending };
});
});
if (r.error) return ['FAIL', r.error];
if (r.error) return ['FAIL', r.error];
@@ -382,22 +423,22 @@ async function checkEvents(ctx) {
@@ -382,22 +423,22 @@ async function checkEvents(ctx) {
async function checkSlot(ctx) {
async function checkSlot(ctx) {
const { opts } = ctx;
const { opts } = ctx;
const id = Number(opts.slot);
const id = Number(opts.slot);
- const live = withDb(ctx.deps, opts.liveDb, (db) => {
- const slot = db.prepare('SELECT id, user_id, slug, title, protocol, streaming_method, ingest_authority, openre_stream_id FROM managed_streams WHERE id = ?').get(id);
+ const live = await withDb(ctx.deps, opts.liveDb, async (db) => {
+ const slot = await db.prepare('SELECT id, user_id, slug, title, protocol, streaming_method, ingest_authority, openre_stream_id FROM managed_streams WHERE id = ?').get(id);
if (!slot) return { slot: null };
if (!slot) return { slot: null };
- const liveNow = db.prepare('SELECT id FROM streams WHERE managed_stream_id = ? AND is_live = 1 LIMIT 1').get(id);
- const subject = db.prepare("SELECT subject_id FROM linked_accounts WHERE service = 'network' AND user_id = ?").get(slot.user_id);
- const user = db.prepare('SELECT username, is_banned FROM users WHERE id = ?').get(slot.user_id);
+ const liveNow = await db.prepare('SELECT id FROM streams WHERE managed_stream_id = ? AND is_live = 1 LIMIT 1').get(id);
+ const subject = await db.prepare("SELECT subject_id FROM linked_accounts WHERE service = 'network' AND user_id = ?").get(slot.user_id);
+ const user = await db.prepare('SELECT username, is_banned FROM users WHERE id = ?').get(slot.user_id);
return { slot, liveNow, subject: subject && subject.subject_id, user };
return { slot, liveNow, subject: subject && subject.subject_id, user };
});
});
if (live.error) return ['FAIL', live.error];
if (live.error) return ['FAIL', live.error];
if (!live.slot) return ['FAIL', `Live has no managed stream ${id}`];
if (!live.slot) return ['FAIL', `Live has no managed stream ${id}`];
const s = live.slot;
const s = live.slot;
- const openre = withDb(ctx.deps, opts.openreDb, (db) => {
- const defs = db.prepare("SELECT d.id, d.state, d.mirror_to_live FROM external_refs r JOIN stream_definitions d ON d.id = r.definition_id WHERE r.service = 'live' AND r.type = 'managed_stream' AND r.ref_id = ?").all(String(id));
- const map = db.prepare("SELECT source_type, status, reason FROM migration_map WHERE source_system = 'live' AND ((source_type = 'managed_stream' AND source_id = ?) OR source_type = 'restream_destination')").all(String(id));
- const dests = defs.length ? db.prepare('SELECT enabled, hold_reason FROM destinations WHERE definition_id = ?').all(defs[0].id) : [];
- const openSessions = defs.length ? db.prepare("SELECT count(*) AS n FROM ingest_sessions WHERE definition_id = ? AND state IN ('starting', 'live', 'ending')").get(defs[0].id).n : 0;
+ const openre = await withDb(ctx.deps, opts.openreDb, async (db) => {
+ const defs = await db.prepare("SELECT d.id, d.state, d.mirror_to_live FROM external_refs r JOIN stream_definitions d ON d.id = r.definition_id WHERE r.service = 'live' AND r.type = 'managed_stream' AND r.ref_id = ?").all(String(id));
+ const map = await db.prepare("SELECT source_type, status, reason FROM migration_map WHERE source_system = 'live' AND ((source_type = 'managed_stream' AND source_id = ?) OR source_type = 'restream_destination')").all(String(id));
+ const dests = defs.length ? await db.prepare('SELECT enabled, hold_reason FROM destinations WHERE definition_id = ?').all(defs[0].id) : [];
+ const openSessions = defs.length ? (await db.prepare("SELECT count(*) AS n FROM ingest_sessions WHERE definition_id = ? AND state IN ('starting', 'live', 'ending')").get(defs[0].id)).n : 0;
return { defs, slotMap: map.find((m) => m.source_type === 'managed_stream') || null, dests, openSessions };
return { defs, slotMap: map.find((m) => m.source_type === 'managed_stream') || null, dests, openSessions };
});
});
if (openre.error) return ['FAIL', openre.error];
if (openre.error) return ['FAIL', openre.error];
@@ -481,4 +522,4 @@ if (require.main === module) {
@@ -481,4 +522,4 @@ if (require.main === module) {
}, (err) => { console.error(`cutover-preflight: ${err.stack || err}`); process.exit(2); });
}, (err) => { console.error(`cutover-preflight: ${err.stack || err}`); process.exit(2); });
}
}
-module.exports = { preflight, parseArgs, format, parseEnvFile, parseSsListeners, localAddresses, rtmpProbe, sameSecret, CHECKS, DEFAULTS };
+module.exports = { preflight, parseArgs, format, parseEnvFile, parseSsListeners, localAddresses, rtmpProbe, sameSecret, withDb, CHECKS, DEFAULTS };
@@ -24,7 +24,7 @@ function arg(name) {
@@ -24,7 +24,7 @@ function arg(name) {
return i > 0 ? process.argv[i + 1] : null;
return i > 0 ? process.argv[i + 1] : null;
}
}
-function main() {
+async function main() {
require('dotenv').config({ path: process.env.OPENRE_ENV_FILE || path.join(process.cwd(), '.env') });
require('dotenv').config({ path: process.env.OPENRE_ENV_FILE || path.join(process.cwd(), '.env') });
const { load } = require('../server/config');
const { load } = require('../server/config');
const { openRuntime } = require('../server/store');
const { openRuntime } = require('../server/store');
@@ -35,9 +35,9 @@ function main() {
@@ -35,9 +35,9 @@ function main() {
const config = load();
const config = load();
if (apply && !config.secretsKey) { console.error('OPENRE_SECRETS_KEY is required to import destinations'); process.exit(2); }
if (apply && !config.secretsKey) { console.error('OPENRE_SECRETS_KEY is required to import destinations'); process.exit(2); }
const liveDb = new Database(liveDbPath, { readonly: true, fileMustExist: true });
const liveDb = new Database(liveDbPath, { readonly: true, fileMustExist: true });
- const rt = openRuntime({ config, log: { log() {}, warn: console.warn, error: console.error } });
+ const rt = await openRuntime({ config, log: { log() {}, warn: console.warn, error: console.error } });
const onlySlots = arg('slots') ? arg('slots').split(',').map(Number).filter(Number.isFinite) : null;
const onlySlots = arg('slots') ? arg('slots').split(',').map(Number).filter(Number.isFinite) : null;
- const report = migrate({ liveDb, rt, apply, onlySlots });
+ const report = await migrate({ liveDb, rt, apply, onlySlots });
const r = config.rtmp;
const r = config.rtmp;
const md = checklist(report, { openreUrl: config.baseUrl, rtmpUrl: `rtmp://${r.publicHost}${r.publicPort === 1935 ? '' : `:${r.publicPort}`}/live` });
const md = checklist(report, { openreUrl: config.baseUrl, rtmpUrl: `rtmp://${r.publicHost}${r.publicPort === 1935 ? '' : `:${r.publicPort}`}/live` });
const out = arg('checklist');
const out = arg('checklist');
new file mode 100755
new file mode 100755
@@ -0,0 +1,27 @@
@@ -0,0 +1,27 @@
+#!/usr/bin/env node
+'use strict';
+/**
+ * The one-time move of OpenRe's SQLite database (OPENRE_DB_PATH) into its PostgreSQL schema (ADR-035; the procedure is
+ * openvibe-sdk docs/migrating-to-postgresql.md, section 6, carried out by openvibe-sdk/db runSqliteMigration).
+ *
+ * node scripts/migrate-to-postgres.js [--sqlite <file>] [--pglite] [--json]
+ *
+ * Applies migrations/ as the owner (DATABASE_DIRECT_URL), copies every table into emptied tables, verifies row counts
+ * and checksums, and exits 1 unless everything verified. The SQLite file is opened read-only. Every table keeps its
+ * name and columns; the SDK outbox's rows move into its PostgreSQL outbox (the same columns). Run with every OpenRe
+ * process stopped (OpenVibe.Host roles/data/switch-service.sh with SWITCH_UNITS).
+ */
+require('dotenv').config({ path: process.env.OPENRE_ENV_FILE || require('path').join(process.cwd(), '.env') });
+const { runSqliteMigration } = require('openvibe-sdk/db');
+const { load } = require('../server/config');
+const { MIGRATIONS } = require('../server/db');
+
+const TABLES = {};
+
+if (require.main === module) {
+ const config = load();
+ runSqliteMigration({ service: 'openre', sqlite: config.dbPath, directUrl: config.db.directUrl, migrations: MIGRATIONS, tables: TABLES })
+ .then((code) => process.exit(code), (err) => { console.error(`migrate-to-postgres failed: ${err.message}`); process.exit(1); });
+}
+
+module.exports = { TABLES };
@@ -10,7 +10,7 @@
@@ -10,7 +10,7 @@
*/
*/
'use strict';
'use strict';
-const { createActorLimiter } = require('openvibe-sdk/limits');
+const { createActorLimiter, createValkeyLimitStore } = require('openvibe-sdk/limits');
const num = (v, d) => { const n = parseInt(v, 10); return Number.isFinite(n) && n > 0 ? n : d; };
const num = (v, d) => { const n = parseInt(v, 10); return Number.isFinite(n) && n > 0 ? n : d; };
@@ -23,7 +23,7 @@ const ROUTES = [
@@ -23,7 +23,7 @@ const ROUTES = [
['openre.stream.create', /^POST$/, /^\/streams\/?$/, { minute: 10, hour: 100 }],
['openre.stream.create', /^POST$/, /^\/streams\/?$/, { minute: 10, hour: 100 }],
];
];
-function createOpenReActorLimits({ env = process.env, registry = null, now } = {}) {
+function createOpenReActorLimits({ env = process.env, registry = null, now, valkey = null } = {}) {
const refused = registry && typeof registry.counter === 'function'
const refused = registry && typeof registry.counter === 'function'
? registry.counter({ name: 'openre_rate_limited_total', help: 'API writes refused 429 by a per-actor limit, by limit name and window', labelNames: ['limit', 'window'] })
? registry.counter({ name: 'openre_rate_limited_total', help: 'API writes refused 429 by a per-actor limit, by limit name and window', labelNames: ['limit', 'window'] })
: null;
: null;
@@ -31,6 +31,8 @@ function createOpenReActorLimits({ env = process.env, registry = null, now } = {
@@ -31,6 +31,8 @@ function createOpenReActorLimits({ env = process.env, registry = null, now } = {
limits: { minute: num(env.OPENRE_LIMITS_MINUTE, 120), hour: num(env.OPENRE_LIMITS_HOUR, 3000) },
limits: { minute: num(env.OPENRE_LIMITS_MINUTE, 120), hour: num(env.OPENRE_LIMITS_HOUR, 3000) },
actor: (req) => (req.caller && req.caller.kind === 'user' && req.caller.subject ? `user:${req.caller.subject}` : null),
actor: (req) => (req.caller && req.caller.kind === 'user' && req.caller.subject ? `user:${req.caller.subject}` : null),
...(now ? { now } : {}),
...(now ? { now } : {}),
+ // Valkey (ADR-035): counters survive a restart of openre-api and are shared if it ever runs twice.
+ ...(valkey ? { store: createValkeyLimitStore(valkey) } : {}),
onLimited(e) {
onLimited(e) {
console.warn(`[Limits] ${e.name}: ${e.actor} refused, over ${e.limit} per ${e.window}`);
console.warn(`[Limits] ${e.name}: ${e.actor} refused, over ${e.limit} per ${e.window}`);
if (refused) refused.inc({ limit: e.name, window: e.window });
if (refused) refused.inc({ limit: e.name, window: e.window });
@@ -46,9 +46,9 @@ function createV1Router({ rt, auth }) {
@@ -46,9 +46,9 @@ function createV1Router({ rt, auth }) {
}
}
};
};
- function publicDefinition(d, { withSessions = true } = {}) {
- const keys = store.definitions.keys(d.id).filter(k => k.status !== 'revoked');
- const open = withSessions ? store.sessions.openFor(d.id)[0] || null : null;
+ async function publicDefinition(d, { withSessions = true } = {}) {
+ const keys = (await store.definitions.keys(d.id)).filter(k => k.status !== 'revoked');
+ const open = withSessions ? (await store.sessions.openFor(d.id))[0] || null : null;
return {
return {
id: d.id,
id: d.id,
title: d.title,
title: d.title,
@@ -62,7 +62,7 @@ function createV1Router({ rt, auth }) {
@@ -62,7 +62,7 @@ function createV1Router({ rt, auth }) {
state: d.state,
state: d.state,
revision: d.revision,
revision: d.revision,
external_refs: d.external_refs,
external_refs: d.external_refs,
- ingest: store.definitions.ingestEndpoints(d),
+ ingest: await store.definitions.ingestEndpoints(d),
keys: keys.map(k => ({ id: k.id, hint: k.hint, status: k.status, grace_until: iso(k.grace_until), created_at: iso(k.created_at), last_used_at: iso(k.last_used_at) })),
keys: keys.map(k => ({ id: k.id, hint: k.hint, status: k.status, grace_until: iso(k.grace_until), created_at: iso(k.created_at), last_used_at: iso(k.last_used_at) })),
session: open ? { id: open.id, state: open.state, protocol: open.protocol, live_at: iso(open.live_at) } : null,
session: open ? { id: open.id, state: open.state, protocol: open.protocol, live_at: iso(open.live_at) } : null,
created_at: iso(d.created_at),
created_at: iso(d.created_at),
@@ -70,7 +70,7 @@ function createV1Router({ rt, auth }) {
@@ -70,7 +70,7 @@ function createV1Router({ rt, auth }) {
};
};
}
}
- function publicSession(s, { detail = false } = {}) {
+ async function publicSession(s, { detail = false } = {}) {
const out = {
const out = {
id: s.id,
id: s.id,
stream_id: s.definition_id,
stream_id: s.definition_id,
@@ -90,17 +90,17 @@ function createV1Router({ rt, auth }) {
@@ -90,17 +90,17 @@ function createV1Router({ rt, auth }) {
duration_seconds: s.live_at ? Math.round(((s.ended_at || Date.now()) - s.live_at) / 1000) : 0,
duration_seconds: s.live_at ? Math.round(((s.ended_at || Date.now()) - s.live_at) / 1000) : 0,
};
};
if (detail) {
if (detail) {
- out.transitions = store.sessions.transitions(s.id).map(t => ({ ...t, at: iso(t.at) }));
- out.outputs = store.outputs.outputsOfSession(s.id);
- const rec = store.recordings.bySession(s.id);
+ out.transitions = (await store.sessions.transitions(s.id)).map(t => ({ ...t, at: iso(t.at) }));
+ out.outputs = await store.outputs.outputsOfSession(s.id);
+ const rec = await store.recordings.bySession(s.id);
out.recording = rec ? { id: rec.id, mode: rec.mode, state: rec.state, media: rec.media_vod_id ? { app: rec.media_app, vod_id: rec.media_vod_id } : null, last_error: rec.last_error } : null;
out.recording = rec ? { id: rec.id, mode: rec.mode, state: rec.state, media: rec.media_vod_id ? { app: rec.media_app, vod_id: rec.media_vod_id } : null, last_error: rec.last_error } : null;
- out.playback = store.sessions.playback(s);
+ out.playback = await store.sessions.playback(s);
}
}
return out;
return out;
}
}
- function loadDefinition(req, res, id) {
- const d = store.definitions.get(id);
+ async function loadDefinition(req, res, id) {
+ const d = await store.definitions.get(id);
if (!d || d.state === 'archived' || !auth.canAccess(req.caller, d.owner_subject)) {
if (!d || d.state === 'archived' || !auth.canAccess(req.caller, d.owner_subject)) {
fail(req, res, 404, 'openre.stream_not_found', 'no such stream definition');
fail(req, res, 404, 'openre.stream_not_found', 'no such stream definition');
return null;
return null;
@@ -108,9 +108,9 @@ function createV1Router({ rt, auth }) {
@@ -108,9 +108,9 @@ function createV1Router({ rt, auth }) {
return d;
return d;
}
}
- function loadDestination(req, res, id) {
- const dest = store.outputs.destinationRow(id);
- const d = dest ? store.definitions.get(dest.definition_id) : null;
+ async function loadDestination(req, res, id) {
+ const dest = await store.outputs.destinationRow(id);
+ const d = dest ? await store.definitions.get(dest.definition_id) : null;
if (!dest || !d || !auth.canAccess(req.caller, d.owner_subject)) {
if (!dest || !d || !auth.canAccess(req.caller, d.owner_subject)) {
fail(req, res, 404, 'openre.destination_not_found', 'no such destination');
fail(req, res, 404, 'openre.destination_not_found', 'no such destination');
return null;
return null;
@@ -118,9 +118,9 @@ function createV1Router({ rt, auth }) {
@@ -118,9 +118,9 @@ function createV1Router({ rt, auth }) {
return dest;
return dest;
}
}
- function loadSession(req, res, id) {
- const s = store.sessions.get(id);
- const d = s ? store.definitions.row(s.definition_id) : null;
+ async function loadSession(req, res, id) {
+ const s = await store.sessions.get(id);
+ const d = s ? await store.definitions.row(s.definition_id) : null;
if (!s || !d || !auth.canAccess(req.caller, d.owner_subject)) {
if (!s || !d || !auth.canAccess(req.caller, d.owner_subject)) {
fail(req, res, 404, 'openre.session_not_found', 'no such session');
fail(req, res, 404, 'openre.session_not_found', 'no such session');
return null;
return null;
@@ -130,29 +130,29 @@ function createV1Router({ rt, auth }) {
@@ -130,29 +130,29 @@ function createV1Router({ rt, auth }) {
// ── Streams ───────────────────────────────────────────────
// ── Streams ───────────────────────────────────────────────
- router.get('/streams', guard('openre.stream.read'), handle((req, res) => {
+ router.get('/streams', guard('openre.stream.read'), handle(async (req, res) => {
const ref = String(req.query.external_ref || '');
const ref = String(req.query.external_ref || '');
if (ref) {
if (ref) {
const [service, type, ...rest] = ref.split(':');
const [service, type, ...rest] = ref.split(':');
- const d = store.definitions.findByRef(service, type, rest.join(':'));
+ const d = await store.definitions.findByRef(service, type, rest.join(':'));
const visible = d && d.state !== 'archived' && auth.canAccess(req.caller, d.owner_subject);
const visible = d && d.state !== 'archived' && auth.canAccess(req.caller, d.owner_subject);
- return res.json({ streams: visible ? [publicDefinition(d)] : [] });
+ return res.json({ streams: visible ? [await publicDefinition(d)] : [] });
}
}
const c = req.caller;
const c = req.caller;
let owner = c.subject || null;
let owner = c.subject || null;
if (c.kind === 'user' && c.staff && req.query.all === '1') owner = null;
if (c.kind === 'user' && c.staff && req.query.all === '1') owner = null;
- const list = store.definitions.list({ owner_subject: owner || undefined, limit: Number(req.query.limit) || 100 });
- return res.json({ streams: list.map(d => publicDefinition(d)) });
+ const list = await store.definitions.list({ owner_subject: owner || undefined, limit: Number(req.query.limit) || 100 });
+ return res.json({ streams: (await Promise.all(list.map(async d => await publicDefinition(d)))) });
}));
}));
- router.post('/streams', guard('openre.stream.write'), handle((req, res) => {
+ router.post('/streams', guard('openre.stream.write'), handle(async (req, res) => {
const owner = req.caller.kind === 'user' ? req.caller.subject : (req.caller.subject || (req.body && req.body.owner_subject));
const owner = req.caller.kind === 'user' ? req.caller.subject : (req.caller.subject || (req.body && req.body.owner_subject));
if (!owner) return fail(req, res, 400, 'openre.invalid_owner', 'name the owner (X-OV-Subject)');
if (!owner) return fail(req, res, 400, 'openre.invalid_owner', 'name the owner (X-OV-Subject)');
if (req.caller.kind === 'service' && req.caller.subject && req.body && req.body.owner_subject && req.body.owner_subject !== req.caller.subject) {
if (req.caller.kind === 'service' && req.caller.subject && req.body && req.body.owner_subject && req.body.owner_subject !== req.caller.subject) {
return fail(req, res, 400, 'openre.invalid_owner', 'owner_subject differs from X-OV-Subject');
return fail(req, res, 400, 'openre.invalid_owner', 'owner_subject differs from X-OV-Subject');
}
}
const b = req.body || {};
const b = req.body || {};
- const { definition, key } = store.definitions.create({
+ const { definition, key } = await store.definitions.create({
owner_subject: owner,
owner_subject: owner,
title: b.title, description: b.description, protocols: b.protocols,
title: b.title, description: b.description, protocols: b.protocols,
recording_mode: b.recording_mode, recording_visibility: b.recording_visibility, playback_visibility: b.playback_visibility,
recording_mode: b.recording_mode, recording_visibility: b.recording_visibility, playback_visibility: b.playback_visibility,
@@ -160,45 +160,45 @@ function createV1Router({ rt, auth }) {
@@ -160,45 +160,45 @@ function createV1Router({ rt, auth }) {
created_by: auth.actorOf(req.caller),
created_by: auth.actorOf(req.caller),
});
});
res.set('Cache-Control', 'no-store');
res.set('Cache-Control', 'no-store');
- return res.status(201).json({ stream: publicDefinition(definition), key: { id: key.id, key: key.key, hint: key.hint, shown_once: true } });
+ return res.status(201).json({ stream: await publicDefinition(definition), key: { id: key.id, key: key.key, hint: key.hint, shown_once: true } });
}));
}));
- router.get('/streams/:id', guard('openre.stream.read'), handle((req, res) => {
- const d = loadDefinition(req, res, req.params.id);
- if (d) res.json({ stream: publicDefinition(d) });
+ router.get('/streams/:id', guard('openre.stream.read'), handle(async (req, res) => {
+ const d = await loadDefinition(req, res, req.params.id);
+ if (d) res.json({ stream: await publicDefinition(d) });
}));
}));
- router.patch('/streams/:id', guard('openre.stream.write'), handle((req, res) => {
- const d = loadDefinition(req, res, req.params.id);
+ router.patch('/streams/:id', guard('openre.stream.write'), handle(async (req, res) => {
+ const d = await loadDefinition(req, res, req.params.id);
if (!d) return;
if (!d) return;
const b = req.body || {};
const b = req.body || {};
- const updated = store.definitions.update(d.id, {
+ const updated = await store.definitions.update(d.id, {
title: b.title, description: b.description, protocols: b.protocols, recording_mode: b.recording_mode,
title: b.title, description: b.description, protocols: b.protocols, recording_mode: b.recording_mode,
recording_visibility: b.recording_visibility, playback_visibility: b.playback_visibility, mirror_to_live: b.mirror_to_live, state: b.state,
recording_visibility: b.recording_visibility, playback_visibility: b.playback_visibility, mirror_to_live: b.mirror_to_live, state: b.state,
});
});
- res.json({ stream: publicDefinition(updated) });
+ res.json({ stream: await publicDefinition(updated) });
}));
}));
- router.delete('/streams/:id', guard('openre.stream.write'), handle((req, res) => {
- const d = loadDefinition(req, res, req.params.id);
+ router.delete('/streams/:id', guard('openre.stream.write'), handle(async (req, res) => {
+ const d = await loadDefinition(req, res, req.params.id);
if (!d) return;
if (!d) return;
- store.definitions.archive(d.id);
+ await store.definitions.archive(d.id);
res.status(204).end();
res.status(204).end();
}));
}));
- router.get('/streams/:id/keys', guard('openre.stream.read'), handle((req, res) => {
- const d = loadDefinition(req, res, req.params.id);
- if (d) res.json({ keys: store.definitions.keys(d.id).map(k => ({ ...k, created_at: iso(k.created_at), grace_until: iso(k.grace_until), revoked_at: iso(k.revoked_at), last_used_at: iso(k.last_used_at) })) });
+ router.get('/streams/:id/keys', guard('openre.stream.read'), handle(async (req, res) => {
+ const d = await loadDefinition(req, res, req.params.id);
+ if (d) res.json({ keys: (await store.definitions.keys(d.id)).map(k => ({ ...k, created_at: iso(k.created_at), grace_until: iso(k.grace_until), revoked_at: iso(k.revoked_at), last_used_at: iso(k.last_used_at) })) });
}));
}));
- router.post('/streams/:id/keys/rotate', guard('openre.key.rotate'), handle((req, res) => {
- const d = loadDefinition(req, res, req.params.id);
+ router.post('/streams/:id/keys/rotate', guard('openre.key.rotate'), handle(async (req, res) => {
+ const d = await loadDefinition(req, res, req.params.id);
if (!d) return;
if (!d) return;
const b = req.body || {};
const b = req.body || {};
- const r = store.definitions.rotateKey(d.id, { grace_seconds: b.grace_seconds, rotated_by: auth.actorOf(req.caller), reason: 'rotated' });
+ const r = await store.definitions.rotateKey(d.id, { grace_seconds: b.grace_seconds, rotated_by: auth.actorOf(req.caller), reason: 'rotated' });
let ended = 0;
let ended = 0;
if (b.end_sessions) {
if (b.end_sessions) {
- for (const s of store.sessions.openFor(d.id)) if (store.sessions.requestEnd(s.id, `key_rotation:${auth.actorOf(req.caller)}`)) ended++;
+ for (const s of await store.sessions.openFor(d.id)) if (await store.sessions.requestEnd(s.id, `key_rotation:${auth.actorOf(req.caller)}`)) ended++;
}
}
res.set('Cache-Control', 'no-store');
res.set('Cache-Control', 'no-store');
res.json({
res.json({
@@ -206,63 +206,63 @@ function createV1Router({ rt, auth }) {
@@ -206,63 +206,63 @@ function createV1Router({ rt, auth }) {
retired: r.retired,
retired: r.retired,
grace_until: iso(r.grace_until),
grace_until: iso(r.grace_until),
sessions_ending: ended,
sessions_ending: ended,
- ingest: store.definitions.ingestEndpoints(store.definitions.get(d.id)),
+ ingest: await store.definitions.ingestEndpoints(await store.definitions.get(d.id)),
});
});
}));
}));
// ── Destinations ──────────────────────────────────────────
// ── Destinations ──────────────────────────────────────────
- router.get('/streams/:id/destinations', guard('openre.output.read'), handle((req, res) => {
- const d = loadDefinition(req, res, req.params.id);
- if (d) res.json({ destinations: store.outputs.destinations(d.id) });
+ router.get('/streams/:id/destinations', guard('openre.output.read'), handle(async (req, res) => {
+ const d = await loadDefinition(req, res, req.params.id);
+ if (d) res.json({ destinations: await store.outputs.destinations(d.id) });
}));
}));
- router.post('/streams/:id/destinations', guard('openre.output.write'), handle((req, res) => {
- const d = loadDefinition(req, res, req.params.id);
- if (d) res.status(201).json({ destination: store.outputs.createDestination(d.id, req.body || {}) });
+ router.post('/streams/:id/destinations', guard('openre.output.write'), handle(async (req, res) => {
+ const d = await loadDefinition(req, res, req.params.id);
+ if (d) res.status(201).json({ destination: await store.outputs.createDestination(d.id, req.body || {}) });
}));
}));
- router.patch('/destinations/:id', guard('openre.output.write'), handle((req, res) => {
- const dest = loadDestination(req, res, req.params.id);
- if (dest) res.json({ destination: store.outputs.updateDestination(dest.id, req.body || {}) });
+ router.patch('/destinations/:id', guard('openre.output.write'), handle(async (req, res) => {
+ const dest = await loadDestination(req, res, req.params.id);
+ if (dest) res.json({ destination: await store.outputs.updateDestination(dest.id, req.body || {}) });
}));
}));
- router.delete('/destinations/:id', guard('openre.output.write'), handle((req, res) => {
- const dest = loadDestination(req, res, req.params.id);
+ router.delete('/destinations/:id', guard('openre.output.write'), handle(async (req, res) => {
+ const dest = await loadDestination(req, res, req.params.id);
if (!dest) return;
if (!dest) return;
- store.outputs.deleteDestination(dest.id);
+ await store.outputs.deleteDestination(dest.id);
res.status(204).end();
res.status(204).end();
}));
}));
router.post('/destinations/:id/test', guard('openre.output.write'), handle(async (req, res) => {
router.post('/destinations/:id/test', guard('openre.output.write'), handle(async (req, res) => {
- const dest = loadDestination(req, res, req.params.id);
+ const dest = await loadDestination(req, res, req.params.id);
if (!dest) return;
if (!dest) return;
const result = await testDestination(dest, { allowPrivate: config.outputs.allowPrivateHosts });
const result = await testDestination(dest, { allowPrivate: config.outputs.allowPrivateHosts });
- store.outputs.log(null, dest.id, result.ok ? 'info' : 'warn', `test ${result.ok ? 'passed' : 'failed'}: ${result.checks.map(c => `${c.check} ${c.ok ? 'ok' : 'FAIL'} (${c.detail})`).join('; ')}`);
+ await store.outputs.log(null, dest.id, result.ok ? 'info' : 'warn', `test ${result.ok ? 'passed' : 'failed'}: ${result.checks.map(c => `${c.check} ${c.ok ? 'ok' : 'FAIL'} (${c.detail})`).join('; ')}`);
res.json(result);
res.json(result);
}));
}));
- router.post('/destinations/:id/start', guard('openre.output.write'), handle((req, res) => {
- const dest = loadDestination(req, res, req.params.id);
- if (dest) res.json({ output: store.outputs.startDestination(dest.id) });
+ router.post('/destinations/:id/start', guard('openre.output.write'), handle(async (req, res) => {
+ const dest = await loadDestination(req, res, req.params.id);
+ if (dest) res.json({ output: await store.outputs.startDestination(dest.id) });
}));
}));
- router.post('/destinations/:id/stop', guard('openre.output.write'), handle((req, res) => {
- const dest = loadDestination(req, res, req.params.id);
- if (dest) res.json({ stopping: store.outputs.stopDestination(dest.id) });
+ router.post('/destinations/:id/stop', guard('openre.output.write'), handle(async (req, res) => {
+ const dest = await loadDestination(req, res, req.params.id);
+ if (dest) res.json({ stopping: await store.outputs.stopDestination(dest.id) });
}));
}));
- router.get('/destinations/:id/logs', guard('openre.output.read'), handle((req, res) => {
- const dest = loadDestination(req, res, req.params.id);
- if (dest) res.json({ logs: store.outputs.logsOfDestination(dest.id, Math.min(Number(req.query.limit) || 100, 500)).map(l => ({ ...l, at: iso(l.at) })) });
+ router.get('/destinations/:id/logs', guard('openre.output.read'), handle(async (req, res) => {
+ const dest = await loadDestination(req, res, req.params.id);
+ if (dest) res.json({ logs: (await store.outputs.logsOfDestination(dest.id, Math.min(Number(req.query.limit) || 100, 500))).map(l => ({ ...l, at: iso(l.at) })) });
}));
}));
// ── Sessions ──────────────────────────────────────────────
// ── Sessions ──────────────────────────────────────────────
- router.get('/sessions', guard('openre.session.read'), handle((req, res) => {
+ router.get('/sessions', guard('openre.session.read'), handle(async (req, res) => {
const q = req.query;
const q = req.query;
if (q.stream_id) {
if (q.stream_id) {
- const d = loadDefinition(req, res, String(q.stream_id));
+ const d = await loadDefinition(req, res, String(q.stream_id));
if (!d) return;
if (!d) return;
}
}
const c = req.caller;
const c = req.caller;
@@ -270,48 +270,48 @@ function createV1Router({ rt, auth }) {
@@ -270,48 +270,48 @@ function createV1Router({ rt, auth }) {
// sessions of the subject it acts for, or every session without X-OV-Subject.
// sessions of the subject it acts for, or every session without X-OV-Subject.
let owner = c.subject || null;
let owner = c.subject || null;
if (c.kind === 'user' && c.staff && q.all === '1') owner = null;
if (c.kind === 'user' && c.staff && q.all === '1') owner = null;
- const list = store.sessions.list({
+ const list = await store.sessions.list({
definition_id: q.stream_id ? String(q.stream_id) : undefined,
definition_id: q.stream_id ? String(q.stream_id) : undefined,
owner_subject: q.stream_id ? undefined : (owner || undefined),
owner_subject: q.stream_id ? undefined : (owner || undefined),
state: q.state ? String(q.state) : undefined,
state: q.state ? String(q.state) : undefined,
limit: q.limit,
limit: q.limit,
before: q.before,
before: q.before,
});
});
- res.json({ sessions: list.map(s => publicSession(s)) });
+ res.json({ sessions: (await Promise.all(list.map(async s => await publicSession(s)))) });
}));
}));
- router.get('/sessions/:id', guard('openre.session.read'), handle((req, res) => {
- const s = loadSession(req, res, req.params.id);
- if (s) res.json({ session: publicSession(s, { detail: true }) });
+ router.get('/sessions/:id', guard('openre.session.read'), handle(async (req, res) => {
+ const s = await loadSession(req, res, req.params.id);
+ if (s) res.json({ session: await publicSession(s, { detail: true }) });
}));
}));
- router.get('/sessions/:id/playback', guard('openre.session.read'), handle((req, res) => {
- const s = loadSession(req, res, req.params.id);
- if (s) res.json({ playback: store.sessions.playback(s) });
+ router.get('/sessions/:id/playback', guard('openre.session.read'), handle(async (req, res) => {
+ const s = await loadSession(req, res, req.params.id);
+ if (s) res.json({ playback: await store.sessions.playback(s) });
}));
}));
- router.post('/sessions/:id/end', guard('openre.session.end'), handle((req, res) => {
- const s = loadSession(req, res, req.params.id);
+ router.post('/sessions/:id/end', guard('openre.session.end'), handle(async (req, res) => {
+ const s = await loadSession(req, res, req.params.id);
if (!s) return;
if (!s) return;
- const requested = store.sessions.requestEnd(s.id, auth.actorOf(req.caller));
- res.status(requested ? 202 : 409).json({ requested, state: store.sessions.get(s.id).state });
+ const requested = await store.sessions.requestEnd(s.id, auth.actorOf(req.caller));
+ res.status(requested ? 202 : 409).json({ requested, state: (await store.sessions.get(s.id)).state });
}));
}));
- router.get('/sessions/:id/outputs', guard('openre.output.read'), handle((req, res) => {
- const s = loadSession(req, res, req.params.id);
- if (s) res.json({ outputs: store.outputs.outputsOfSession(s.id) });
+ router.get('/sessions/:id/outputs', guard('openre.output.read'), handle(async (req, res) => {
+ const s = await loadSession(req, res, req.params.id);
+ if (s) res.json({ outputs: await store.outputs.outputsOfSession(s.id) });
}));
}));
- router.get('/outputs/:id/logs', guard('openre.output.read'), handle((req, res) => {
- const o = store.outputs.outputRow(req.params.id);
- const s = o ? loadSession(req, res, o.session_id) : null;
+ router.get('/outputs/:id/logs', guard('openre.output.read'), handle(async (req, res) => {
+ const o = await store.outputs.outputRow(req.params.id);
+ const s = o ? await loadSession(req, res, o.session_id) : null;
if (!o) return fail(req, res, 404, 'openre.output_not_found', 'no such output');
if (!o) return fail(req, res, 404, 'openre.output_not_found', 'no such output');
- if (s) res.json({ logs: store.outputs.logsOfOutput(o.id, Math.min(Number(req.query.limit) || 100, 500)).map(l => ({ ...l, at: iso(l.at) })) });
+ if (s) res.json({ logs: (await store.outputs.logsOfOutput(o.id, Math.min(Number(req.query.limit) || 100, 500))).map(l => ({ ...l, at: iso(l.at) })) });
}));
}));
- router.get('/workers', guard('openre.session.read'), handle((req, res) => {
+ router.get('/workers', guard('openre.session.read'), handle(async (req, res) => {
if (req.caller.kind === 'user' && !req.caller.staff) return fail(req, res, 403, 'capability.denied', 'staff only');
if (req.caller.kind === 'user' && !req.caller.staff) return fail(req, res, 403, 'capability.denied', 'staff only');
- res.json({ workers: store.workers.recent(100).map(w => ({ ...w, started_at: iso(w.started_at), ready_at: iso(w.ready_at), heartbeat_at: iso(w.heartbeat_at), drain_started_at: iso(w.drain_started_at), drain_deadline: iso(w.drain_deadline), stopped_at: iso(w.stopped_at) })) });
+ res.json({ workers: (await store.workers.recent(100)).map(w => ({ ...w, started_at: iso(w.started_at), ready_at: iso(w.ready_at), heartbeat_at: iso(w.heartbeat_at), drain_started_at: iso(w.drain_started_at), drain_deadline: iso(w.drain_deadline), stopped_at: iso(w.stopped_at) })) });
}));
}));
return { router, publicDefinition, publicSession };
return { router, publicDefinition, publicSession };
@@ -22,7 +22,7 @@ function createApp({ rt, auth, keys, log = console, fetchImpl }) {
@@ -22,7 +22,7 @@ function createApp({ rt, auth, keys, log = console, fetchImpl }) {
// sessions by state, from openvibe-shared/metrics (mounted before every route).
// sessions by state, from openvibe-shared/metrics (mounted before every route).
const metrics = require('openvibe-shared/metrics').instrument(app, { service: 'openre' });
const metrics = require('openvibe-shared/metrics').instrument(app, { service: 'openre' });
metrics.registry.gauge({ name: 'openre_sessions', help: 'Ingest sessions by state', labelNames: ['state'],
metrics.registry.gauge({ name: 'openre_sessions', help: 'Ingest sessions by state', labelNames: ['state'],
- collect: () => db.prepare("SELECT state, count(*) AS n FROM ingest_sessions WHERE state IN ('starting','live') GROUP BY state").all().map((r) => ({ labels: { state: r.state }, value: r.n })) });
+ collect: async () => (await db.prepare("SELECT state, count(*) AS n FROM ingest_sessions WHERE state IN ('starting','live') GROUP BY state").all()).map((r) => ({ labels: { state: r.state }, value: r.n })) });
app.use(contracts.http.middleware());
app.use(contracts.http.middleware());
// One W3C trace across services (openvibe-shared/trace): calls made while serving a request carry its traceparent.
// One W3C trace across services (openvibe-shared/trace): calls made while serving a request carry its traceparent.
require('openvibe-shared/trace').install(app);
require('openvibe-shared/trace').install(app);
@@ -64,46 +64,50 @@ function createApp({ rt, auth, keys, log = console, fetchImpl }) {
@@ -64,46 +64,50 @@ function createApp({ rt, auth, keys, log = console, fetchImpl }) {
// serving reads, rather than failing it. The worker list, the lease and the outbox stay in the body for
// serving reads, rather than failing it. The worker list, the lease and the outbox stay in the body for
// scripts/cutover-preflight.js; a restore drill never publishes, so its relay check is skipped, never ok.
// scripts/cutover-preflight.js; a restore drill never publishes, so its relay check is skipped, never ok.
const { createReadiness, skip } = require('openvibe-shared/ready');
const { createReadiness, skip } = require('openvibe-shared/ready');
- const workerView = () => store.workers.alive().map(w => ({ kind: w.kind, generation: w.generation, state: w.state, heartbeat_age_ms: Date.now() - w.heartbeat_at }));
- const leaseView = () => {
- const l = db.prepare("SELECT holder, expires_at FROM leases WHERE name = 'coordinator'").get();
+ const workerView = async () => (await store.workers.alive()).map(w => ({ kind: w.kind, generation: w.generation, state: w.state, heartbeat_age_ms: Date.now() - w.heartbeat_at }));
+ const leaseView = async () => {
+ const l = await db.prepare("SELECT holder, expires_at FROM leases WHERE name = 'coordinator'").get();
return l ? { holder: l.holder, lease_valid: l.expires_at > Date.now() } : null;
return l ? { holder: l.holder, lease_valid: l.expires_at > Date.now() } : null;
};
};
+ // Valkey (ADR-035): the per-actor limit counters (optional: without VALKEY_URL they count in this process).
+ const valkey = config.valkey.url ? require('openvibe-sdk/valkey').createValkey({ url: config.valkey.url, prefix: config.valkey.prefix }) : null;
+ app.locals.valkey = valkey;
const readiness = createReadiness({
const readiness = createReadiness({
service: 'openre',
service: 'openre',
release: release.release,
release: release.release,
checks: [
checks: [
- { name: 'db', required: true, description: 'SQLite answers a query', check: () => db.prepare('SELECT 1 AS ok').get().ok === 1 },
+ { name: 'db', required: true, description: 'SQLite answers a query', check: async () => (await db.prepare('SELECT 1 AS ok').get()).ok === 1 },
{ name: 'network_key', required: true, description: 'OpenVibe.Network RS256 key (sign-in and service tokens)', check: () => keys.loaded() || 'Network public key not loaded yet' },
{ name: 'network_key', required: true, description: 'OpenVibe.Network RS256 key (sign-in and service tokens)', check: () => keys.loaded() || 'Network public key not loaded yet' },
{
{
name: 'workers', required: false, description: 'a ready rtmp-ingest and restream worker',
name: 'workers', required: false, description: 'a ready rtmp-ingest and restream worker',
- check: () => {
- const alive = workerView();
+ check: async () => {
+ const alive = await workerView();
const missing = WORKER_KINDS.filter(k => !alive.some(w => w.kind === k && w.state === 'ready'));
const missing = WORKER_KINDS.filter(k => !alive.some(w => w.kind === k && w.state === 'ready'));
return missing.length ? `no ready ${missing.join(' or ')} worker` : { ok: true, detail: { ready: alive.filter(w => w.state === 'ready').map(w => `${w.kind}#${w.generation}`) } };
return missing.length ? `no ready ${missing.join(' or ')} worker` : { ok: true, detail: { ready: alive.filter(w => w.state === 'ready').map(w => `${w.kind}#${w.generation}`) } };
},
},
},
},
- { name: 'coordinator', required: false, description: 'the coordinator holds a valid lease', check: () => { const l = leaseView(); return (l && l.lease_valid) || 'no valid coordinator lease'; } },
+ { name: 'valkey', required: false, description: 'per-actor limit counters', check: async () => (valkey ? await valkey.ready() : skip('VALKEY_URL not set: limits count in this process')) },
+ { name: 'coordinator', required: false, description: 'the coordinator holds a valid lease', check: async () => { const l = await leaseView(); return (l && l.lease_valid) || 'no valid coordinator lease'; } },
{
{
name: 'events_relay', required: false, description: 'durable events to OpenVibe.Events',
name: 'events_relay', required: false, description: 'durable events to OpenVibe.Events',
- check: () => {
- const st = events.status();
+ check: async () => {
+ const st = await events.status();
if (!st.configured) return config.drill ? skip('restore drill: never publishes') : 'relay off (EVENTS_URL or OV_OAUTH_CLIENT_SECRET unset): openre.* events wait in the outbox';
if (!st.configured) return config.drill ? skip('restore drill: never publishes') : 'relay off (EVENTS_URL or OV_OAUTH_CLIENT_SECRET unset): openre.* events wait in the outbox';
return { ok: true, detail: { pending: st.pending, rejected: st.rejected } };
return { ok: true, detail: { pending: st.pending, rejected: st.rejected } };
},
},
},
},
],
],
- details: (body) => {
+ details: async (body) => {
let workers = [];
let workers = [];
let coordinator = null;
let coordinator = null;
- try { workers = workerView(); coordinator = leaseView(); } catch { /* reported as empty */ }
+ try { workers = await workerView(); coordinator = await leaseView(); } catch { /* reported as empty */ }
const dbOk = body.checks.db && body.checks.db.status === 'ok';
const dbOk = body.checks.db && body.checks.db.status === 'ok';
return {
return {
...(config.drill ? { mode: 'drill' } : {}),
...(config.drill ? { mode: 'drill' } : {}),
- store: { engine: 'sqlite', path_configured: Boolean(config.dbPath) },
+ store: { engine: 'postgresql' },
workers,
workers,
coordinator,
coordinator,
- events: dbOk ? events.status() : null,
+ events: dbOk ? await events.status() : null,
};
};
},
},
});
});
@@ -112,17 +116,17 @@ function createApp({ rt, auth, keys, log = console, fetchImpl }) {
@@ -112,17 +116,17 @@ function createApp({ rt, auth, keys, log = console, fetchImpl }) {
const apiJson = express.json({ limit: '256kb', type: ['application/json', 'application/*+json'] });
const apiJson = express.json({ limit: '256kb', type: ['application/json', 'application/*+json'] });
const v1 = createV1Router({ rt, auth });
const v1 = createV1Router({ rt, auth });
// Per-actor limits on a signed-in person's writes (server/api/actor-limits.js; roadmap WS-R task 4).
// Per-actor limits on a signed-in person's writes (server/api/actor-limits.js; roadmap WS-R task 4).
- const actorLimits = require('./api/actor-limits').createOpenReActorLimits({ registry: metrics.registry });
+ const actorLimits = require('./api/actor-limits').createOpenReActorLimits({ registry: metrics.registry, valkey });
app.use('/api/v1', apiJson, auth.middleware({ services: true }), actorLimits, v1.router);
app.use('/api/v1', apiJson, auth.middleware({ services: true }), actorLimits, v1.router);
// Public HTTP-FLV playback of a live session, proxied from the worker that holds it. A viewer
// Public HTTP-FLV playback of a live session, proxied from the worker that holds it. A viewer
// deploy of this API interrupts viewers of this URL (they reconnect), never the ingest.
// deploy of this API interrupts viewers of this URL (they reconnect), never the ingest.
- app.get('/play/:file', (req, res) => {
+ app.get('/play/:file', async (req, res) => {
const m = SESSION_FLV_RE.exec(req.params.file);
const m = SESSION_FLV_RE.exec(req.params.file);
- const s = m ? store.sessions.get(m[1]) : null;
- const d = s ? store.definitions.row(s.definition_id) : null;
+ const s = m ? await store.sessions.get(m[1]) : null;
+ const d = s ? await store.definitions.row(s.definition_id) : null;
if (!s || !d || s.state !== 'live' || d.playback_visibility === 'private') return res.status(404).end();
if (!s || !d || s.state !== 'live' || d.playback_visibility === 'private') return res.status(404).end();
- const pb = store.sessions.playback(s);
+ const pb = await store.sessions.playback(s);
if (!pb || !pb.flv) return res.status(404).end();
if (!pb || !pb.flv) return res.status(404).end();
const upstream = http.get(pb.flv.internal_url, (up) => {
const upstream = http.get(pb.flv.internal_url, (up) => {
if (up.statusCode !== 200) { res.status(502).end(); up.resume(); return; }
if (up.statusCode !== 200) { res.status(502).end(); up.resume(); return; }
@@ -54,7 +54,7 @@ function createKeyStore({ urls = [], pem = null, fetchImpl = globalThis.fetch, l
@@ -54,7 +54,7 @@ function createKeyStore({ urls = [], pem = null, fetchImpl = globalThis.fetch, l
return null;
return null;
}
}
- function start() {
+ async function start() {
if (pem) return Promise.resolve(key);
if (pem) return Promise.resolve(key);
const attempt = async () => {
const attempt = async () => {
const k = await fetchOnce();
const k = await fetchOnce();
@@ -63,7 +63,7 @@ function createKeyStore({ urls = [], pem = null, fetchImpl = globalThis.fetch, l
@@ -63,7 +63,7 @@ function createKeyStore({ urls = [], pem = null, fetchImpl = globalThis.fetch, l
};
};
refreshTimer = setInterval(() => { fetchOnce().catch(() => {}); }, 6 * 60 * 60 * 1000);
refreshTimer = setInterval(() => { fetchOnce().catch(() => {}); }, 6 * 60 * 60 * 1000);
refreshTimer.unref?.();
refreshTimer.unref?.();
- return attempt();
+ return await attempt();
}
}
return { get: () => key, loaded: () => Boolean(key), start, stop() { clearTimeout(retryTimer); clearInterval(refreshTimer); }, fetchOnce };
return { get: () => key, loaded: () => Boolean(key), start, stop() { clearTimeout(retryTimer); clearInterval(refreshTimer); }, fetchOnce };
@@ -57,6 +57,11 @@ function load(env = process.env) {
@@ -57,6 +57,11 @@ function load(env = process.env) {
trustProxy: env.TRUST_PROXY != null ? Number(env.TRUST_PROXY) : 2,
trustProxy: env.TRUST_PROXY != null ? Number(env.TRUST_PROXY) : 2,
// ── Store ───────────────────────────────────────────────
// ── Store ───────────────────────────────────────────────
+ // PostgreSQL (ADR-035): DATABASE_URL serves through PgBouncer, DATABASE_DIRECT_URL (the owner) runs migrations/.
+ // Without DATABASE_URL, development and tests use an embedded PGlite database (OPENRE_PGLITE_DIR). dbPath: the
+ // SQLite file the one-time move reads (scripts/migrate-to-postgres.js).
+ db: { url: env.DATABASE_URL || '', directUrl: env.DATABASE_DIRECT_URL || '', pgliteDir: env.OPENRE_PGLITE_DIR || '' },
+ valkey: { url: env.VALKEY_URL || '', prefix: env.VALKEY_PREFIX || 'ov:openre:' },
dbPath: env.OPENRE_DB_PATH || './data/openre.db',
dbPath: env.OPENRE_DB_PATH || './data/openre.db',
// ── Secrets ─────────────────────────────────────────────
// ── Secrets ─────────────────────────────────────────────
@@ -33,59 +33,59 @@ function createCoordinator({ rt, media, lineage = null, log = console, holder =
@@ -33,59 +33,59 @@ function createCoordinator({ rt, media, lineage = null, log = console, holder =
let ticking = null;
let ticking = null;
const stats = { ticks: 0, lastTickAt: null, lastError: null, holdsLease: false };
const stats = { ticks: 0, lastTickAt: null, lastError: null, holdsLease: false };
- function takeLease() {
- const ok = lease.take.run({ holder, exp: now() + config.workers.leaseMs, now: now() }).changes > 0;
+ async function takeLease() {
+ const ok = (await lease.take.run({ holder, exp: now() + config.workers.leaseMs, now: now() })).changes > 0;
stats.holdsLease = ok;
stats.holdsLease = ok;
return ok;
return ok;
}
}
- function failSessionsOf(worker, reason) {
+ async function failSessionsOf(worker, reason) {
let n = 0;
let n = 0;
- for (const s of store.sessions.ofWorker(worker.id)) {
- const r = store.sessions.transition(s.id, 'failed', { reason, actor: 'coordinator' });
+ for (const s of await store.sessions.ofWorker(worker.id)) {
+ const r = await store.sessions.transition(s.id, 'failed', { reason, actor: 'coordinator' });
if (r.ok) n++;
if (r.ok) n++;
}
}
return n;
return n;
}
}
/** The synchronous part of a tick (everything but Media calls and the relay). */
/** The synchronous part of a tick (everything but Media calls and the relay). */
- function syncTick() {
+ async function syncTick() {
const out = { lost: 0, failed: 0, drained: 0, outputsCreated: 0, outputsAssigned: 0, keysRevoked: 0, recordingsCreated: 0 };
const out = { lost: 0, failed: 0, drained: 0, outputsCreated: 0, outputsAssigned: 0, keysRevoked: 0, recordingsCreated: 0 };
// 2. liveness
// 2. liveness
- for (const w of store.workers.stale(now() - config.workers.leaseMs)) {
- if (store.workers.lose(w.id, 'heartbeat_missed')) {
+ for (const w of await store.workers.stale(now() - config.workers.leaseMs)) {
+ if (await store.workers.lose(w.id, 'heartbeat_missed')) {
out.lost++;
out.lost++;
- out.failed += failSessionsOf(w, 'worker_lost');
- store.outputs.release(w.id);
+ out.failed += await failSessionsOf(w, 'worker_lost');
+ await store.outputs.release(w.id);
log.warn(`[coordinator] worker ${w.kind}#${w.generation} (${w.id}) lost; sessions failed, outputs released`);
log.warn(`[coordinator] worker ${w.kind}#${w.generation} (${w.id}) lost; sessions failed, outputs released`);
}
}
}
}
// 3. orphans (lease expired, worker not alive)
// 3. orphans (lease expired, worker not alive)
- for (const s of store.sessions.expiredLeases()) {
- const w = s.worker_id ? store.workers.get(s.worker_id) : null;
+ for (const s of await store.sessions.expiredLeases()) {
+ const w = s.worker_id ? await store.workers.get(s.worker_id) : null;
if (w && ['starting', 'ready', 'draining'].includes(w.state) && w.heartbeat_at >= now() - config.workers.leaseMs) continue;
if (w && ['starting', 'ready', 'draining'].includes(w.state) && w.heartbeat_at >= now() - config.workers.leaseMs) continue;
- if (store.sessions.transition(s.id, 'failed', { reason: 'lease_expired', actor: 'coordinator' }).ok) out.failed++;
+ if ((await store.sessions.transition(s.id, 'failed', { reason: 'lease_expired', actor: 'coordinator' })).ok) out.failed++;
}
}
// 4. generations
// 4. generations
for (const kind of KINDS) {
for (const kind of KINDS) {
- const newest = store.workers.newestReady(kind);
+ const newest = await store.workers.newestReady(kind);
if (!newest) continue;
if (!newest) continue;
- for (const old of store.workers.olderReady(kind, newest.generation)) {
- if (store.workers.drain(old.id, now() + config.workers.drainMaxMs)) {
+ for (const old of await store.workers.olderReady(kind, newest.generation)) {
+ if (await store.workers.drain(old.id, now() + config.workers.drainMaxMs)) {
out.drained++;
out.drained++;
log.log(`[coordinator] ${kind}#${old.generation} drains (newest ready is #${newest.generation})`);
log.log(`[coordinator] ${kind}#${old.generation} drains (newest ready is #${newest.generation})`);
}
}
}
}
}
}
// 5. outputs
// 5. outputs
- out.outputsCreated = store.outputs.ensureAutoOutputs();
- const restream = store.workers.newestReady('restream');
- if (restream) out.outputsAssigned = store.outputs.assignPending(restream);
- store.outputs.stopOrphans();
+ out.outputsCreated = await store.outputs.ensureAutoOutputs();
+ const restream = await store.workers.newestReady('restream');
+ if (restream) out.outputsAssigned = await store.outputs.assignPending(restream);
+ await store.outputs.stopOrphans();
// 6. keys
// 6. keys
- out.keysRevoked = store.definitions.expireGraceKeys();
+ out.keysRevoked = await store.definitions.expireGraceKeys();
// 7. recordings (requests only; the Media calls are async below)
// 7. recordings (requests only; the Media calls are async below)
- if (media && media.configured) out.recordingsCreated = store.recordings.ensureRequests();
+ if (media && media.configured) out.recordingsCreated = await store.recordings.ensureRequests();
return out;
return out;
}
}
@@ -93,8 +93,8 @@ function createCoordinator({ rt, media, lineage = null, log = console, holder =
@@ -93,8 +93,8 @@ function createCoordinator({ rt, media, lineage = null, log = console, holder =
if (ticking) return ticking;
if (ticking) return ticking;
ticking = (async () => {
ticking = (async () => {
try {
try {
- if (!takeLease()) return { skipped: 'lease held by another coordinator' };
- const out = syncTick();
+ if (!await takeLease()) return { skipped: 'lease held by another coordinator' };
+ const out = await syncTick();
if (media) out.recordingSteps = await store.recordings.process(media);
if (media) out.recordingSteps = await store.recordings.process(media);
if (lineage) out.lineage = await lineage.refresh();
if (lineage) out.lineage = await lineage.refresh();
stats.ticks++;
stats.ticks++;
@@ -124,7 +124,7 @@ function createCoordinator({ rt, media, lineage = null, log = console, holder =
@@ -124,7 +124,7 @@ function createCoordinator({ rt, media, lineage = null, log = console, holder =
clearTimeout(timer);
clearTimeout(timer);
if (ticking) await ticking.catch(() => {});
if (ticking) await ticking.catch(() => {});
await rt.events.stop();
await rt.events.stop();
- lease.release.run(holder);
+ await lease.release.run(holder);
},
},
stats: () => ({ ...stats, lineage: lineage ? lineage.stats() : null }),
stats: () => ({ ...stats, lineage: lineage ? lineage.stats() : null }),
};
};
@@ -1,248 +1,42 @@
@@ -1,248 +1,42 @@
'use strict';
'use strict';
/**
/**
- * OpenRe's durable store: one SQLite database (WAL) shared by every OpenRe process.
+ * OpenRe's durable store: one PostgreSQL database (ov_openre, ADR-035) shared by every OpenRe process.
*
*
* openre-api definitions, keys, destinations (owner writes)
* openre-api definitions, keys, destinations (owner writes)
* openre-session-coordinator worker liveness, drain, output assignment, recordings, event relay
* openre-session-coordinator worker liveness, drain, output assignment, recordings, event relay
* openre-rtmp-ingest sessions it admits, their leases and transitions
* openre-rtmp-ingest sessions it admits, their leases and transitions
* openre-restream-worker output state, health and logs
* openre-restream-worker output state, health and logs
*
*
- * ADR-007: SQLite stays the documented store while the platform is single-host. Several writer
- * processes is the exact trigger ADR-007 names for moving to PostgreSQL; every statement lives
- * behind server/store/* so that move changes one layer. WAL + busy_timeout keeps the writers
- * safe today (each write is one short transaction).
- *
- * Every table is created idempotently on open. Times are epoch milliseconds.
+ * Several writer processes were the trigger ADR-007 named for leaving SQLite; every statement lives behind
+ * server/store/*, and every transaction there is SERIALIZABLE (store/index.js), as SQLite's one writer made them.
+ * The schema is migrations/NNNN_*.sql, applied at boot by the owner role. Times are epoch milliseconds.
*/
*/
const fs = require('fs');
const fs = require('fs');
const path = require('path');
const path = require('path');
-const Database = require('better-sqlite3');
-
-const SCHEMA = `
-CREATE TABLE IF NOT EXISTS stream_definitions (
- id TEXT PRIMARY KEY,
- owner_subject TEXT NOT NULL,
- title TEXT NOT NULL DEFAULT 'Untitled stream',
- description TEXT NOT NULL DEFAULT '',
- protocols TEXT NOT NULL DEFAULT '["rtmp"]',
- recording_mode TEXT NOT NULL DEFAULT 'vod' CHECK (recording_mode IN ('vod', 'clips', 'none')),
- recording_visibility TEXT NOT NULL DEFAULT 'public' CHECK (recording_visibility IN ('public', 'unlisted', 'private')),
- playback_visibility TEXT NOT NULL DEFAULT 'public' CHECK (playback_visibility IN ('public', 'unlisted', 'private')),
- mirror_to_live INTEGER NOT NULL DEFAULT 0,
- state TEXT NOT NULL DEFAULT 'active' CHECK (state IN ('active', 'disabled', 'archived')),
- revision INTEGER NOT NULL DEFAULT 1,
- created_at INTEGER NOT NULL,
- updated_at INTEGER NOT NULL
-);
-CREATE INDEX IF NOT EXISTS idx_definitions_owner ON stream_definitions(owner_subject, state);
-
-CREATE TABLE IF NOT EXISTS external_refs (
- definition_id TEXT NOT NULL REFERENCES stream_definitions(id) ON DELETE CASCADE,
- service TEXT NOT NULL,
- type TEXT NOT NULL,
- ref_id TEXT NOT NULL,
- label TEXT,
- created_at INTEGER NOT NULL,
- PRIMARY KEY (definition_id, service, type, ref_id)
-);
-CREATE INDEX IF NOT EXISTS idx_external_refs_ref ON external_refs(service, type, ref_id);
-
-CREATE TABLE IF NOT EXISTS definition_lineage (
- definition_id TEXT PRIMARY KEY REFERENCES stream_definitions(id) ON DELETE CASCADE,
- resolution TEXT NOT NULL,
- checked_at INTEGER NOT NULL
-);
-
-CREATE TABLE IF NOT EXISTS ingest_keys (
- id TEXT PRIMARY KEY,
- definition_id TEXT NOT NULL REFERENCES stream_definitions(id) ON DELETE CASCADE,
- key_hash TEXT NOT NULL UNIQUE,
- hint TEXT NOT NULL,
- status TEXT NOT NULL CHECK (status IN ('active', 'grace', 'revoked')),
- created_by TEXT,
- created_at INTEGER NOT NULL,
- grace_until INTEGER,
- revoked_at INTEGER,
- revoked_reason TEXT,
- last_used_at INTEGER
-);
-CREATE INDEX IF NOT EXISTS idx_ingest_keys_definition ON ingest_keys(definition_id, status);
-
-CREATE TABLE IF NOT EXISTS workers (
- id TEXT PRIMARY KEY,
- kind TEXT NOT NULL,
- generation INTEGER NOT NULL,
- release TEXT,
- pid INTEGER,
- host TEXT,
- state TEXT NOT NULL CHECK (state IN ('starting', 'ready', 'draining', 'stopped', 'lost')),
- endpoints TEXT NOT NULL DEFAULT '{}',
- started_at INTEGER NOT NULL,
- ready_at INTEGER,
- heartbeat_at INTEGER NOT NULL,
- drain_started_at INTEGER,
- drain_deadline INTEGER,
- stopped_at INTEGER,
- stop_reason TEXT,
- UNIQUE (kind, generation)
-);
-CREATE INDEX IF NOT EXISTS idx_workers_kind_state ON workers(kind, state, generation);
-
-CREATE TABLE IF NOT EXISTS ingest_sessions (
- id TEXT PRIMARY KEY,
- definition_id TEXT NOT NULL REFERENCES stream_definitions(id),
- key_id TEXT,
- protocol TEXT NOT NULL,
- state TEXT NOT NULL CHECK (state IN ('starting', 'live', 'ending', 'ended', 'failed')),
- worker_id TEXT,
- worker_kind TEXT,
- worker_generation INTEGER,
- lease_expires_at INTEGER,
- desired_state TEXT NOT NULL DEFAULT 'run' CHECK (desired_state IN ('run', 'end')),
- end_requested_by TEXT,
- end_reason TEXT,
- failure_reason TEXT,
- media_info TEXT,
- revision INTEGER NOT NULL DEFAULT 1,
- created_at INTEGER NOT NULL,
- live_at INTEGER,
- ending_at INTEGER,
- ended_at INTEGER,
- updated_at INTEGER NOT NULL
-);
-CREATE INDEX IF NOT EXISTS idx_sessions_definition ON ingest_sessions(definition_id, created_at);
-CREATE INDEX IF NOT EXISTS idx_sessions_state ON ingest_sessions(state);
-CREATE INDEX IF NOT EXISTS idx_sessions_worker ON ingest_sessions(worker_id, state);
-
-CREATE TABLE IF NOT EXISTS session_transitions (
- id INTEGER PRIMARY KEY AUTOINCREMENT,
- session_id TEXT NOT NULL REFERENCES ingest_sessions(id) ON DELETE CASCADE,
- from_state TEXT,
- to_state TEXT NOT NULL,
- reason TEXT,
- actor TEXT,
- at INTEGER NOT NULL
-);
-CREATE INDEX IF NOT EXISTS idx_transitions_session ON session_transitions(session_id, id);
+const { createDb } = require('openvibe-sdk/db');
-CREATE TABLE IF NOT EXISTS destinations (
- id TEXT PRIMARY KEY,
- definition_id TEXT NOT NULL REFERENCES stream_definitions(id) ON DELETE CASCADE,
- platform TEXT NOT NULL CHECK (platform IN ('youtube', 'twitch', 'kick', 'custom')),
- name TEXT,
- server_url TEXT NOT NULL,
- stream_key_enc TEXT,
- key_hint TEXT,
- srt_passphrase_enc TEXT,
- srt_latency_ms INTEGER,
- enabled INTEGER NOT NULL DEFAULT 1,
- auto_start INTEGER NOT NULL DEFAULT 1,
- quality_preset TEXT NOT NULL DEFAULT 'auto',
- custom_video_bitrate INTEGER,
- custom_audio_bitrate INTEGER,
- custom_fps INTEGER,
- custom_encoder_preset TEXT,
- hold_reason TEXT,
- consecutive_failures INTEGER NOT NULL DEFAULT 0,
- cooldown_until INTEGER,
- last_error TEXT,
- last_failed_at INTEGER,
- created_at INTEGER NOT NULL,
- updated_at INTEGER NOT NULL
-);
-CREATE INDEX IF NOT EXISTS idx_destinations_definition ON destinations(definition_id);
+const MIGRATIONS = path.join(__dirname, '..', 'migrations');
+const DEV_PGLITE = path.join(__dirname, '..', 'data', 'pglite');
-CREATE TABLE IF NOT EXISTS outputs (
- id TEXT PRIMARY KEY,
- session_id TEXT NOT NULL REFERENCES ingest_sessions(id),
- destination_id TEXT NOT NULL REFERENCES destinations(id) ON DELETE CASCADE,
- desired TEXT NOT NULL DEFAULT 'run' CHECK (desired IN ('run', 'stop')),
- state TEXT NOT NULL DEFAULT 'pending' CHECK (state IN ('pending', 'starting', 'live', 'error', 'failed', 'stopped')),
- worker_id TEXT,
- worker_generation INTEGER,
- restart_attempts INTEGER NOT NULL DEFAULT 0,
- next_restart_at INTEGER,
- ever_live INTEGER NOT NULL DEFAULT 0,
- last_error TEXT,
- progress TEXT,
- started_at INTEGER,
- live_at INTEGER,
- ended_at INTEGER,
- created_at INTEGER NOT NULL,
- updated_at INTEGER NOT NULL,
- UNIQUE (session_id, destination_id)
-);
-CREATE INDEX IF NOT EXISTS idx_outputs_worker ON outputs(worker_id, state);
-CREATE INDEX IF NOT EXISTS idx_outputs_session ON outputs(session_id);
-
-CREATE TABLE IF NOT EXISTS output_logs (
- id INTEGER PRIMARY KEY AUTOINCREMENT,
- output_id TEXT,
- destination_id TEXT NOT NULL,
- level TEXT NOT NULL CHECK (level IN ('info', 'warn', 'error')),
- message TEXT NOT NULL,
- at INTEGER NOT NULL
-);
-CREATE INDEX IF NOT EXISTS idx_output_logs_output ON output_logs(output_id, id);
-CREATE INDEX IF NOT EXISTS idx_output_logs_destination ON output_logs(destination_id, id);
-
-CREATE TABLE IF NOT EXISTS recordings (
- id TEXT PRIMARY KEY,
- session_id TEXT NOT NULL UNIQUE REFERENCES ingest_sessions(id),
- mode TEXT NOT NULL CHECK (mode IN ('vod', 'clips')),
- state TEXT NOT NULL CHECK (state IN ('pending', 'requested', 'recording', 'finalizing', 'finalized', 'failed', 'cancelled')),
- media_app TEXT,
- media_vod_id TEXT,
- attempts INTEGER NOT NULL DEFAULT 0,
- next_attempt_at INTEGER NOT NULL DEFAULT 0,
- last_error TEXT,
- created_at INTEGER NOT NULL,
- updated_at INTEGER NOT NULL
-);
-CREATE INDEX IF NOT EXISTS idx_recordings_state ON recordings(state, next_attempt_at);
-
-CREATE TABLE IF NOT EXISTS migration_map (
- source_system TEXT NOT NULL,
- source_type TEXT NOT NULL,
- source_id TEXT NOT NULL,
- target_type TEXT,
- target_id TEXT,
- status TEXT NOT NULL CHECK (status IN ('imported', 'held', 'excluded')),
- reason TEXT,
- imported_at INTEGER NOT NULL,
- PRIMARY KEY (source_system, source_type, source_id)
-);
-
-CREATE TABLE IF NOT EXISTS leases (
- name TEXT PRIMARY KEY,
- holder TEXT NOT NULL,
- expires_at INTEGER NOT NULL
-);
-`;
-
-function openDb(dbPath) {
- if (dbPath !== ':memory:') fs.mkdirSync(path.dirname(path.resolve(dbPath)), { recursive: true });
- const db = new Database(dbPath);
- // busy_timeout first: several OpenRe processes open the database at the same time on boot,
- // and every statement after this one (the WAL switch, the schema) may have to wait for a lock.
- db.pragma('busy_timeout = 5000');
- db.pragma('journal_mode = WAL');
- db.pragma('foreign_keys = ON');
- db.pragma('synchronous = NORMAL');
- db.exec(SCHEMA);
- return db;
+/**
+ * The serving handle: DATABASE_URL through PgBouncer, after the migrations ran as the owner (DATABASE_DIRECT_URL).
+ * Without DATABASE_URL (development, tests), an embedded PGlite database: in OPENRE_PGLITE_DIR or data/pglite, or in
+ * memory with { memory: true }. PGlite is one process: run the other OpenRe processes against PostgreSQL.
+ */
+async function openDb(config, { memory = false, log = console, registry } = {}) {
+ const quiet = { ...log, info() {}, log() {} };
+ if (!config.db.url) {
+ if (config.isProduction) throw new Error('DATABASE_URL is not set: production serves from PostgreSQL (OpenVibe.Host roles/data add-service.sh openre)');
+ const dir = memory ? null : (config.db.pgliteDir || DEV_PGLITE);
+ if (dir) fs.mkdirSync(dir, { recursive: true });
+ const db = createDb({ pglite: dir || true, service: 'openre', registry, log });
+ await db.migrate({ dir: MIGRATIONS, log: quiet });
+ return db;
+ }
+ if (!config.db.directUrl) throw new Error('DATABASE_DIRECT_URL is not set: migrations run with the owner role on a direct connection');
+ const owner = createDb({ url: config.db.directUrl, service: 'openre-migrate', max: 1, log });
+ try { await owner.migrate({ dir: MIGRATIONS, log: quiet }); } finally { await owner.close(); }
+ return createDb({ url: config.db.url, service: 'openre', registry, log });
}
}
-module.exports = { openDb, SCHEMA };
+module.exports = { openDb, MIGRATIONS };
@@ -2,7 +2,7 @@
@@ -2,7 +2,7 @@
/**
/**
* OpenRe → OpenVibe.Events through the openvibe-sdk transactional outbox (ADR-004).
* OpenRe → OpenVibe.Events through the openvibe-sdk transactional outbox (ADR-004).
*
*
- * Every OpenRe process can enqueue (the row commits in the same SQLite transaction as the change
+ * Every OpenRe process can enqueue (the row commits in the same transaction as the change
* it describes); only the session coordinator runs the relay that publishes rows with OpenRe's
* it describes); only the session coordinator runs the relay that publishes rows with OpenRe's
* service token (audience openvibe.events, capability events.event.publish). Events down, Network
* service token (audience openvibe.events, capability events.event.publish). Events down, Network
* down or no credentials yet: rows wait and are retried; nothing in the transport path waits.
* down or no credentials yet: rows wait and are retried; nothing in the transport path waits.
@@ -19,7 +19,7 @@
@@ -19,7 +19,7 @@
*/
*/
const { createClient } = require('openvibe-sdk/core');
const { createClient } = require('openvibe-sdk/core');
const { createServiceTokenClient } = require('openvibe-sdk/auth');
const { createServiceTokenClient } = require('openvibe-sdk/auth');
-const { createEventsClient, createOutbox } = require('openvibe-sdk/events');
+const { createEventsClient, createPgOutbox } = require('openvibe-sdk/events');
const TYPES = Object.freeze({
const TYPES = Object.freeze({
sessionStarted: 'openre.session.started',
sessionStarted: 'openre.session.started',
@@ -58,7 +58,7 @@ function createEvents({ config, db, fetchImpl = globalThis.fetch, now = () => Da
@@ -58,7 +58,7 @@ function createEvents({ config, db, fetchImpl = globalThis.fetch, now = () => Da
});
});
const eventsClient = createEventsClient(client, { source: 'openre' });
const eventsClient = createEventsClient(client, { source: 'openre' });
let lastError = null;
let lastError = null;
- const outbox = createOutbox(db, {
+ const outbox = createPgOutbox(db, {
events: eventsClient,
events: eventsClient,
intervalMs: config.events.intervalMs,
intervalMs: config.events.intervalMs,
now,
now,
@@ -68,15 +68,15 @@ function createEvents({ config, db, fetchImpl = globalThis.fetch, now = () => Da
@@ -68,15 +68,15 @@ function createEvents({ config, db, fetchImpl = globalThis.fetch, now = () => Da
lastError = msg;
lastError = msg;
},
},
});
});
- outbox.ensureSchema();
return {
return {
outbox,
outbox,
configured,
configured,
- enqueue: (envelope) => outbox.enqueue(envelope),
+ // Joins the caller's transaction (the SDK handle is ambient): the row commits with the change it describes.
+ enqueue: async (envelope) => await outbox.enqueue(db, envelope),
start() { if (configured) outbox.start(); return configured; },
start() { if (configured) outbox.start(); return configured; },
stop: () => outbox.stop(),
stop: () => outbox.stop(),
- status: () => ({ configured, pending: outbox.pending(), rejected: outbox.rejected(), last_error: lastError }),
+ status: async () => ({ configured, pending: await outbox.pending(), rejected: await outbox.rejected(), last_error: lastError }),
};
};
}
}
@@ -6,7 +6,7 @@
@@ -6,7 +6,7 @@
* end requests) and serves the UI. Restarting or redeploying it never touches a worker process,
* end requests) and serves the UI. Restarting or redeploying it never touches a worker process,
* a session or an ffmpeg output; see test/api-restart.test.js.
* a session or an ffmpeg output; see test/api-restart.test.js.
*
*
- * start() is what the tests use too: it takes a config plus injectable clock/fetch/log.
+ * start() is what the tests use too: it takes a config plus injectable clock/fetch/log (and a database handle).
*/
*/
const path = require('path');
const path = require('path');
const { load } = require('./config');
const { load } = require('./config');
@@ -14,9 +14,9 @@ const { openRuntime } = require('./store');
@@ -14,9 +14,9 @@ const { openRuntime } = require('./store');
const { createKeyStore, createAuth } = require('./auth');
const { createKeyStore, createAuth } = require('./auth');
const { createApp } = require('./app');
const { createApp } = require('./app');
-async function start({ config, clock, fetchImpl = globalThis.fetch, log = console, listen = true } = {}) {
+async function start({ config, clock, fetchImpl = globalThis.fetch, log = console, listen = true, db } = {}) {
config = config || load();
config = config || load();
- const rt = openRuntime({ config, clock, fetchImpl, log });
+ const rt = await openRuntime({ config, clock, fetchImpl, log, db });
const keys = createKeyStore({ urls: [config.networkInternalUrl, config.networkUrl], pem: config.networkPublicKey, fetchImpl, log });
const keys = createKeyStore({ urls: [config.networkInternalUrl, config.networkUrl], pem: config.networkPublicKey, fetchImpl, log });
const auth = createAuth({ config, keys });
const auth = createAuth({ config, keys });
const app = createApp({ rt, auth, keys, log, fetchImpl });
const app = createApp({ rt, auth, keys, log, fetchImpl });
@@ -37,7 +37,8 @@ async function start({ config, clock, fetchImpl = globalThis.fetch, log = consol
@@ -37,7 +37,8 @@ async function start({ config, clock, fetchImpl = globalThis.fetch, log = consol
server.closeAllConnections?.();
server.closeAllConnections?.();
await new Promise(resolve => server.close(() => resolve()));
await new Promise(resolve => server.close(() => resolve()));
}
}
- rt.db.close();
+ if (app.locals.valkey) await app.locals.valkey.close().catch(() => {});
+ await rt.db.close();
}
}
return { config, rt, keys, keyLoaded, auth, app, server, close };
return { config, rt, keys, keyLoaded, auth, app, server, close };
@@ -43,10 +43,10 @@ function createLineage({ db, config, env = process.env, fetchImpl = globalThis.f
@@ -43,10 +43,10 @@ function createLineage({ db, config, env = process.env, fetchImpl = globalThis.f
async function refresh() {
async function refresh() {
if (!enabled) return { skipped: 'off' };
if (!enabled) return { skipped: 'off' };
const out = { checked: 0, resolved: 0 };
const out = { checked: 0, resolved: 0 };
- for (const def of q.due.all(now() - maxAgeMs, batch)) {
+ for (const def of await q.due.all(now() - maxAgeMs, batch)) {
try {
try {
const r = await resolveOne(def);
const r = await resolveOne(def);
- q.save.run(def.id, JSON.stringify(r), now());
+ await q.save.run(def.id, JSON.stringify(r), now());
out.checked++; stats.refreshed++;
out.checked++; stats.refreshed++;
if (r.status === 'resolved') { out.resolved++; stats.resolved++; } else stats.unresolved++;
if (r.status === 'resolved') { out.resolved++; stats.resolved++; } else stats.unresolved++;
} catch (err) {
} catch (err) {
@@ -56,10 +56,10 @@ function createMediaClient({ config, fetchImpl = globalThis.fetch }) {
@@ -56,10 +56,10 @@ function createMediaClient({ config, fetchImpl = globalThis.fetch }) {
return {
return {
configured,
configured,
appId: config.media.appId,
appId: config.media.appId,
- createVod: (fields) => request('POST', '/vods', { body: fields }),
- ingestRtmp: (vodId, rtmpUrl) => request('POST', `/vods/${encodeURIComponent(vodId)}/ingest/rtmp`, { body: { rtmp_url: rtmpUrl } }),
- finalizeVod: (vodId) => request('POST', `/vods/${encodeURIComponent(vodId)}/finalize`, { timeoutMs: 30000 }),
- deleteVod: (vodId) => request('DELETE', `/vods/${encodeURIComponent(vodId)}`),
+ createVod: async (fields) => await request('POST', '/vods', { body: fields }),
+ ingestRtmp: async (vodId, rtmpUrl) => await request('POST', `/vods/${encodeURIComponent(vodId)}/ingest/rtmp`, { body: { rtmp_url: rtmpUrl } }),
+ finalizeVod: async (vodId) => await request('POST', `/vods/${encodeURIComponent(vodId)}/finalize`, { timeoutMs: 30000 }),
+ deleteVod: async (vodId) => await request('DELETE', `/vods/${encodeURIComponent(vodId)}`),
};
};
}
}
@@ -18,8 +18,8 @@
@@ -18,8 +18,8 @@
const { validateDestinationUrl } = require('./destination-url');
const { validateDestinationUrl } = require('./destination-url');
-function columns(db, table) {
- try { return new Set(db.prepare(`PRAGMA table_info(${table})`).all().map(c => c.name)); } catch { return new Set(); }
+async function columns(db, table) {
+ try { return new Set((await db.prepare(`PRAGMA table_info(${table})`).all()).map(c => c.name)); } catch { return new Set(); }
}
}
function recordingModeOf(ms, channel) {
function recordingModeOf(ms, channel) {
@@ -38,21 +38,21 @@ function visibilityOf(ms, channel) {
@@ -38,21 +38,21 @@ function visibilityOf(ms, channel) {
* migrate({ liveDb, rt, apply, onlySlots }) -> report
* migrate({ liveDb, rt, apply, onlySlots }) -> report
* report = { slots: [...], destinations: [...], channels: Map username → {...}, counts }
* report = { slots: [...], destinations: [...], channels: Map username → {...}, counts }
*/
*/
-function migrate({ liveDb, rt, apply = false, onlySlots = null, now = () => Date.now() }) {
+async function migrate({ liveDb, rt, apply = false, onlySlots = null, now = () => Date.now() }) {
const { db, store, config } = rt;
const { db, store, config } = rt;
- const msCols = columns(liveDb, 'managed_streams');
+ const msCols = await columns(liveDb, 'managed_streams');
if (!msCols.size) throw new Error('the Live snapshot has no managed_streams table');
if (!msCols.size) throw new Error('the Live snapshot has no managed_streams table');
- const rdCols = columns(liveDb, 'restream_destinations');
- const laCols = columns(liveDb, 'linked_accounts');
- const chCols = columns(liveDb, 'channels');
+ const rdCols = await columns(liveDb, 'restream_destinations');
+ const laCols = await columns(liveDb, 'linked_accounts');
+ const chCols = await columns(liveDb, 'channels');
const mapGet = db.prepare("SELECT * FROM migration_map WHERE source_system = 'live' AND source_type = ? AND source_id = ?");
const mapGet = db.prepare("SELECT * FROM migration_map WHERE source_system = 'live' AND source_type = ? AND source_id = ?");
const mapPut = db.prepare(`INSERT INTO migration_map (source_system, source_type, source_id, target_type, target_id, status, reason, imported_at)
const mapPut = db.prepare(`INSERT INTO migration_map (source_system, source_type, source_id, target_type, target_id, status, reason, imported_at)
VALUES ('live', @source_type, @source_id, @target_type, @target_id, @status, @reason, @at)
VALUES ('live', @source_type, @source_id, @target_type, @target_id, @status, @reason, @at)
ON CONFLICT(source_system, source_type, source_id) DO UPDATE SET target_type = @target_type, target_id = @target_id, status = @status, reason = @reason, imported_at = @at`);
ON CONFLICT(source_system, source_type, source_id) DO UPDATE SET target_type = @target_type, target_id = @target_id, status = @status, reason = @reason, imported_at = @at`);
- const record = (row) => { if (apply) mapPut.run({ target_type: null, target_id: null, reason: null, ...row, at: now() }); };
+ const record = async (row) => { if (apply) await mapPut.run({ target_type: null, target_id: null, reason: null, ...row, at: now() }); };
- const slots = liveDb.prepare(`SELECT ms.*, u.username, u.display_name, u.is_banned FROM managed_streams ms JOIN users u ON u.id = ms.user_id ORDER BY ms.user_id, ms.id`).all()
+ const slots = (await liveDb.prepare(`SELECT ms.*, u.username, u.display_name, u.is_banned FROM managed_streams ms JOIN users u ON u.id = ms.user_id ORDER BY ms.user_id, ms.id`).all())
.filter(ms => !onlySlots || onlySlots.includes(Number(ms.id)));
.filter(ms => !onlySlots || onlySlots.includes(Number(ms.id)));
const subjectOf = laCols.has('subject_id')
const subjectOf = laCols.has('subject_id')
? liveDb.prepare("SELECT subject_id FROM linked_accounts WHERE service = 'network' AND user_id = ? AND subject_id IS NOT NULL")
? liveDb.prepare("SELECT subject_id FROM linked_accounts WHERE service = 'network' AND user_id = ? AND subject_id IS NOT NULL")
@@ -73,11 +73,11 @@ function migrate({ liveDb, rt, apply = false, onlySlots = null, now = () => Date
@@ -73,11 +73,11 @@ function migrate({ liveDb, rt, apply = false, onlySlots = null, now = () => Date
for (const ms of slots) {
for (const ms of slots) {
const ch = channel(ms);
const ch = channel(ms);
- const prior = mapGet.get('managed_stream', String(ms.id));
+ const prior = await mapGet.get('managed_stream', String(ms.id));
const entry = { live_id: ms.id, slug: ms.slug || null, title: ms.title, protocol: ms.protocol, streaming_method: ms.streaming_method || null, definition_id: null, status: null, reason: null, destinations: [] };
const entry = { live_id: ms.id, slug: ms.slug || null, title: ms.title, protocol: ms.protocol, streaming_method: ms.streaming_method || null, definition_id: null, status: null, reason: null, destinations: [] };
ch.slots.push(entry);
ch.slots.push(entry);
- if (prior && prior.status === 'imported' && store.definitions.get(prior.target_id)) {
+ if (prior && prior.status === 'imported' && await store.definitions.get(prior.target_id)) {
entry.definition_id = prior.target_id;
entry.definition_id = prior.target_id;
entry.status = 'imported';
entry.status = 'imported';
entry.reason = 'already imported';
entry.reason = 'already imported';
@@ -87,12 +87,12 @@ function migrate({ liveDb, rt, apply = false, onlySlots = null, now = () => Date
@@ -87,12 +87,12 @@ function migrate({ liveDb, rt, apply = false, onlySlots = null, now = () => Date
if (!subject) {
if (!subject) {
entry.status = 'held';
entry.status = 'held';
entry.reason = `Live user ${ms.user_id} has no canonical subject yet (it is recorded when they next sign in to Live); the slot is imported on a later run`;
entry.reason = `Live user ${ms.user_id} has no canonical subject yet (it is recorded when they next sign in to Live); the slot is imported on a later run`;
- record({ source_type: 'managed_stream', source_id: String(ms.id), status: 'held', reason: entry.reason });
+ await record({ source_type: 'managed_stream', source_id: String(ms.id), status: 'held', reason: entry.reason });
count('held');
count('held');
} else if (ms.is_banned) {
} else if (ms.is_banned) {
entry.status = 'excluded';
entry.status = 'excluded';
entry.reason = 'the Live account is banned';
entry.reason = 'the Live account is banned';
- record({ source_type: 'managed_stream', source_id: String(ms.id), status: 'excluded', reason: entry.reason });
+ await record({ source_type: 'managed_stream', source_id: String(ms.id), status: 'excluded', reason: entry.reason });
count('excluded');
count('excluded');
} else {
} else {
const chRow = channelOf ? channelOf.get(ms.user_id) : null;
const chRow = channelOf ? channelOf.get(ms.user_id) : null;
@@ -113,10 +113,10 @@ function migrate({ liveDb, rt, apply = false, onlySlots = null, now = () => Date
@@ -113,10 +113,10 @@ function migrate({ liveDb, rt, apply = false, onlySlots = null, now = () => Date
};
};
entry.recording_mode = fields.recording_mode;
entry.recording_mode = fields.recording_mode;
if (apply) {
if (apply) {
- const existing = store.definitions.findByRef('live', 'managed_stream', String(ms.id));
- const def = existing || store.definitions.create(fields).definition; // the new key is discarded unseen
+ const existing = await store.definitions.findByRef('live', 'managed_stream', String(ms.id));
+ const def = existing || (await store.definitions.create(fields)).definition; // the new key is discarded unseen
entry.definition_id = def.id;
entry.definition_id = def.id;
- record({ source_type: 'managed_stream', source_id: String(ms.id), target_type: 'stream_definition', target_id: def.id, status: 'imported', reason: 'new key issued, old key not imported' });
+ await record({ source_type: 'managed_stream', source_id: String(ms.id), target_type: 'stream_definition', target_id: def.id, status: 'imported', reason: 'new key issued, old key not imported' });
}
}
entry.status = 'imported';
entry.status = 'imported';
entry.reason = 'new key issued, old key not imported';
entry.reason = 'new key issued, old key not imported';
@@ -127,28 +127,28 @@ function migrate({ liveDb, rt, apply = false, onlySlots = null, now = () => Date
@@ -127,28 +127,28 @@ function migrate({ liveDb, rt, apply = false, onlySlots = null, now = () => Date
// Destinations: the slot's own, plus unbound ones when the user has exactly one slot.
// Destinations: the slot's own, plus unbound ones when the user has exactly one slot.
const own = destsOfSlot ? destsOfSlot.all(ms.id) : [];
const own = destsOfSlot ? destsOfSlot.all(ms.id) : [];
const unbound = unboundOfUser ? unboundOfUser.all(ms.user_id) : [];
const unbound = unboundOfUser ? unboundOfUser.all(ms.user_id) : [];
- const oneSlot = slotsOfUser.get(ms.user_id).n === 1;
+ const oneSlot = (await slotsOfUser.get(ms.user_id)).n === 1;
const candidates = [...own.map(d => ({ d, unbound: false })), ...unbound.map(d => ({ d, unbound: true }))];
const candidates = [...own.map(d => ({ d, unbound: false })), ...unbound.map(d => ({ d, unbound: true }))];
for (const { d, unbound: isUnbound } of candidates) {
for (const { d, unbound: isUnbound } of candidates) {
const di = { live_id: d.id, platform: d.platform, name: d.name || d.platform, status: null, reason: null, destination_id: null, oauth_linked: Boolean(d.connection_id) };
const di = { live_id: d.id, platform: d.platform, name: d.name || d.platform, status: null, reason: null, destination_id: null, oauth_linked: Boolean(d.connection_id) };
entry.destinations.push(di);
entry.destinations.push(di);
report.destinations.push(di);
report.destinations.push(di);
- const priorD = mapGet.get('restream_destination', String(d.id));
- if (priorD && priorD.target_id && store.outputs.destinationRow(priorD.target_id)) {
+ const priorD = await mapGet.get('restream_destination', String(d.id));
+ if (priorD && priorD.target_id && await store.outputs.destinationRow(priorD.target_id)) {
di.status = priorD.status; di.reason = priorD.reason || 'already imported'; di.destination_id = priorD.target_id; count('skipped');
di.status = priorD.status; di.reason = priorD.reason || 'already imported'; di.destination_id = priorD.target_id; count('skipped');
continue;
continue;
}
}
if (isUnbound && !oneSlot) {
if (isUnbound && !oneSlot) {
di.status = 'held';
di.status = 'held';
di.reason = 'not bound to a slot and the user has several slots (Live pushed unbound destinations from every slot); bind it in Live or add it on openre.stream';
di.reason = 'not bound to a slot and the user has several slots (Live pushed unbound destinations from every slot); bind it in Live or add it on openre.stream';
- record({ source_type: 'restream_destination', source_id: String(d.id), status: 'held', reason: di.reason });
+ await record({ source_type: 'restream_destination', source_id: String(d.id), status: 'held', reason: di.reason });
count('held');
count('held');
continue;
continue;
}
}
if (entry.status !== 'imported') {
if (entry.status !== 'imported') {
di.status = 'held';
di.status = 'held';
di.reason = `its slot is ${entry.status}`;
di.reason = `its slot is ${entry.status}`;
- record({ source_type: 'restream_destination', source_id: String(d.id), status: 'held', reason: di.reason });
+ await record({ source_type: 'restream_destination', source_id: String(d.id), status: 'held', reason: di.reason });
count('held');
count('held');
continue;
continue;
}
}
@@ -174,12 +174,12 @@ function migrate({ liveDb, rt, apply = false, onlySlots = null, now = () => Date
@@ -174,12 +174,12 @@ function migrate({ liveDb, rt, apply = false, onlySlots = null, now = () => Date
if (hold) {
if (hold) {
// Keep the (sealed) key so the owner only has to fix the URL; the row stays
// Keep the (sealed) key so the owner only has to fix the URL; the row stays
// disabled until they do. The original URL is kept in the hold reason.
// disabled until they do. The original URL is kept in the hold reason.
- created = store.outputs.createDestination(entry.definition_id, { ...input, server_url: 'rtmp://held.invalid/app', enabled: false }, { hold_reason: `${hold} (was ${String(d.server_url || '').replace(/\/[^/]*$/, '/…')})` });
+ created = await store.outputs.createDestination(entry.definition_id, { ...input, server_url: 'rtmp://held.invalid/app', enabled: false }, { hold_reason: `${hold} (was ${String(d.server_url || '').replace(/\/[^/]*$/, '/…')})` });
} else {
} else {
- created = store.outputs.createDestination(entry.definition_id, input);
+ created = await store.outputs.createDestination(entry.definition_id, input);
}
}
di.destination_id = created.id;
di.destination_id = created.id;
- record({ source_type: 'restream_destination', source_id: String(d.id), target_type: 'destination', target_id: created.id, status: hold ? 'held' : 'imported', reason: hold || null });
+ await record({ source_type: 'restream_destination', source_id: String(d.id), target_type: 'destination', target_id: created.id, status: hold ? 'held' : 'imported', reason: hold || null });
}
}
di.status = hold ? 'held' : 'imported';
di.status = hold ? 'held' : 'imported';
di.reason = hold;
di.reason = hold;
@@ -71,34 +71,34 @@ function createDefinitions({ db, config, events, clock }) {
@@ -71,34 +71,34 @@ function createDefinitions({ db, config, events, clock }) {
expiredGrace: db.prepare("SELECT id FROM ingest_keys WHERE status = 'grace' AND grace_until <= ?"),
expiredGrace: db.prepare("SELECT id FROM ingest_keys WHERE status = 'grace' AND grace_until <= ?"),
};
};
- function row(id) {
- const d = q.get.get(id);
+ async function row(id) {
+ const d = await q.get.get(id);
if (!d) return null;
if (!d) return null;
return {
return {
...d,
...d,
protocols: parseJson(d.protocols, ['rtmp']),
protocols: parseJson(d.protocols, ['rtmp']),
mirror_to_live: Boolean(d.mirror_to_live),
mirror_to_live: Boolean(d.mirror_to_live),
- external_refs: q.refs.all(id),
+ external_refs: await q.refs.all(id),
};
};
}
}
- function get(id) {
- return isId('stream', id) ? row(id) : null;
+ async function get(id) {
+ return isId('stream', id) ? await row(id) : null;
}
}
- function list({ owner_subject, state, limit = 100 } = {}) {
+ async function list({ owner_subject, state, limit = 100 } = {}) {
const where = [];
const where = [];
const args = [];
const args = [];
if (owner_subject) { where.push('owner_subject = ?'); args.push(owner_subject); }
if (owner_subject) { where.push('owner_subject = ?'); args.push(owner_subject); }
if (state) { where.push('state = ?'); args.push(state); } else where.push("state != 'archived'");
if (state) { where.push('state = ?'); args.push(state); } else where.push("state != 'archived'");
- const rows = db.prepare(`SELECT id FROM stream_definitions ${where.length ? `WHERE ${where.join(' AND ')}` : ''}
+ const rows = await db.prepare(`SELECT id FROM stream_definitions ${where.length ? `WHERE ${where.join(' AND ')}` : ''}
ORDER BY created_at DESC LIMIT ?`).all(...args, Math.min(Math.max(limit, 1), 500));
ORDER BY created_at DESC LIMIT ?`).all(...args, Math.min(Math.max(limit, 1), 500));
- return rows.map(r => row(r.id));
+ return (await Promise.all(rows.map(async r => await row(r.id))));
}
}
- function findByRef(service, type, refId) {
- const r = q.byRef.get(String(service), String(type), String(refId));
- return r ? row(r.definition_id) : null;
+ async function findByRef(service, type, refId) {
+ const r = await q.byRef.get(String(service), String(type), String(refId));
+ return r ? await row(r.definition_id) : null;
}
}
function normalize(input, base = {}) {
function normalize(input, base = {}) {
@@ -125,10 +125,10 @@ function createDefinitions({ db, config, events, clock }) {
@@ -125,10 +125,10 @@ function createDefinitions({ db, config, events, clock }) {
}
}
/** Issue a new active key (inside the caller's transaction). Returns the plain key once. */
/** Issue a new active key (inside the caller's transaction). Returns the plain key once. */
- function issueKey(definitionId, createdBy) {
+ async function issueKey(definitionId, createdBy) {
const plain = newIngestKey();
const plain = newIngestKey();
const id = newId('key', now());
const id = newId('key', now());
- q.insertKey.run(id, definitionId, hashIngestKey(plain), hintOf(plain), createdBy || null, now());
+ await q.insertKey.run(id, definitionId, hashIngestKey(plain), hintOf(plain), createdBy || null, now());
return { id, key: plain, hint: hintOf(plain) };
return { id, key: plain, hint: hintOf(plain) };
}
}
@@ -136,47 +136,47 @@ function createDefinitions({ db, config, events, clock }) {
@@ -136,47 +136,47 @@ function createDefinitions({ db, config, events, clock }) {
* create({ owner_subject, title, ..., external_refs, created_by }) -> { definition, key: { id, key, hint } }
* create({ owner_subject, title, ..., external_refs, created_by }) -> { definition, key: { id, key, hint } }
* The first ingest key is issued with the definition; its plain value is only in this return.
* The first ingest key is issued with the definition; its plain value is only in this return.
*/
*/
- function create(input) {
+ async function create(input) {
if (!ids.isSubjectId('user', input.owner_subject)) throw new StoreError(400, 'openre.invalid_owner', 'owner_subject must be a usr_… subject id');
if (!ids.isSubjectId('user', input.owner_subject)) throw new StoreError(400, 'openre.invalid_owner', 'owner_subject must be a usr_… subject id');
const given = Object.fromEntries(Object.entries(input).filter(([, v]) => v !== undefined && v !== null && v !== ''));
const given = Object.fromEntries(Object.entries(input).filter(([, v]) => v !== undefined && v !== null && v !== ''));
const fields = normalize({ protocols: ['rtmp'], recording_mode: 'vod', recording_visibility: 'public', playback_visibility: 'public', mirror_to_live: false, ...given });
const fields = normalize({ protocols: ['rtmp'], recording_mode: 'vod', recording_visibility: 'public', playback_visibility: 'public', mirror_to_live: false, ...given });
const refs = validateRefs(input.external_refs);
const refs = validateRefs(input.external_refs);
const id = newId('stream', now());
const id = newId('stream', now());
- return db.transaction(() => {
+ return await db.tx(async () => {
for (const r of refs) {
for (const r of refs) {
if (SHARED_REF_TYPES.has(r.type)) continue;
if (SHARED_REF_TYPES.has(r.type)) continue;
- const taken = q.byRef.get(r.service, r.type, r.id);
- if (taken && row(taken.definition_id).state !== 'archived') throw new StoreError(409, 'openre.ref_taken', `${r.service}:${r.type}:${r.id} already belongs to another stream definition`);
+ const taken = await q.byRef.get(r.service, r.type, r.id);
+ if (taken && (await row(taken.definition_id)).state !== 'archived') throw new StoreError(409, 'openre.ref_taken', `${r.service}:${r.type}:${r.id} already belongs to another stream definition`);
}
}
- q.insert.run({ id, owner_subject: input.owner_subject, description: '', ...fields, now: now() });
- for (const r of refs) q.insertRef.run(id, r.service, r.type, r.id, r.label, now());
- const key = issueKey(id, input.created_by);
- return { definition: row(id), key };
- })();
+ await q.insert.run({ id, owner_subject: input.owner_subject, description: '', ...fields, now: now() });
+ for (const r of refs) await q.insertRef.run(id, r.service, r.type, r.id, r.label, now());
+ const key = await issueKey(id, input.created_by);
+ return { definition: await row(id), key };
+ });
}
}
- function update(id, input) {
- const current = get(id);
+ async function update(id, input) {
+ const current = await get(id);
if (!current || current.state === 'archived') throw new StoreError(404, 'openre.stream_not_found', 'no such stream definition');
if (!current || current.state === 'archived') throw new StoreError(404, 'openre.stream_not_found', 'no such stream definition');
const fields = normalize(input, current);
const fields = normalize(input, current);
const keys = Object.keys(fields);
const keys = Object.keys(fields);
if (!keys.length) return current;
if (!keys.length) return current;
- db.prepare(`UPDATE stream_definitions SET ${keys.map(k => `${k} = @${k}`).join(', ')}, revision = revision + 1, updated_at = @now WHERE id = @id`)
+ await db.prepare(`UPDATE stream_definitions SET ${keys.map(k => `${k} = @${k}`).join(', ')}, revision = revision + 1, updated_at = @now WHERE id = @id`)
.run({ ...fields, id, now: now() });
.run({ ...fields, id, now: now() });
- return row(id);
+ return await row(id);
}
}
- function archive(id) {
- const current = get(id);
+ async function archive(id) {
+ const current = await get(id);
if (!current || current.state === 'archived') throw new StoreError(404, 'openre.stream_not_found', 'no such stream definition');
if (!current || current.state === 'archived') throw new StoreError(404, 'openre.stream_not_found', 'no such stream definition');
- const live = db.prepare("SELECT id FROM ingest_sessions WHERE definition_id = ? AND state IN ('starting', 'live', 'ending')").get(id);
+ const live = await db.prepare("SELECT id FROM ingest_sessions WHERE definition_id = ? AND state IN ('starting', 'live', 'ending')").get(id);
if (live) throw new StoreError(409, 'openre.stream_live', 'end the live session before archiving this stream');
if (live) throw new StoreError(409, 'openre.stream_live', 'end the live session before archiving this stream');
- db.transaction(() => {
- db.prepare("UPDATE stream_definitions SET state = 'archived', revision = revision + 1, updated_at = ? WHERE id = ?").run(now(), id);
- for (const k of q.activeKeys.all(id)) q.revokeKey.run(now(), 'archived', k.id);
- db.prepare("UPDATE ingest_keys SET status = 'revoked', revoked_at = ?, revoked_reason = 'archived' WHERE definition_id = ? AND status = 'grace'").run(now(), id);
- })();
- return row(id);
+ await db.tx(async () => {
+ await db.prepare("UPDATE stream_definitions SET state = 'archived', revision = revision + 1, updated_at = ? WHERE id = ?").run(now(), id);
+ for (const k of await q.activeKeys.all(id)) await q.revokeKey.run(now(), 'archived', k.id);
+ await db.prepare("UPDATE ingest_keys SET status = 'revoked', revoked_at = ?, revoked_reason = 'archived' WHERE definition_id = ? AND status = 'grace'").run(now(), id);
+ });
+ return await row(id);
}
}
/**
/**
@@ -185,19 +185,19 @@ function createDefinitions({ db, config, events, clock }) {
@@ -185,19 +185,19 @@ function createDefinitions({ db, config, events, clock }) {
* coordinator when the grace ends). Sessions already running are never cut by a rotation
* coordinator when the grace ends). Sessions already running are never cut by a rotation
* unless end_sessions is set (the caller then asks the owning worker to end them).
* unless end_sessions is set (the caller then asks the owning worker to end them).
*/
*/
- function rotateKey(id, { grace_seconds = 0, rotated_by = null, reason = 'rotated' } = {}) {
- const current = get(id);
+ async function rotateKey(id, { grace_seconds = 0, rotated_by = null, reason = 'rotated' } = {}) {
+ const current = await get(id);
if (!current || current.state === 'archived') throw new StoreError(404, 'openre.stream_not_found', 'no such stream definition');
if (!current || current.state === 'archived') throw new StoreError(404, 'openre.stream_not_found', 'no such stream definition');
const grace = Math.max(0, Math.min(Number(grace_seconds) || 0, 7 * 24 * 3600));
const grace = Math.max(0, Math.min(Number(grace_seconds) || 0, 7 * 24 * 3600));
- return db.transaction(() => {
- const old = q.activeKeys.all(id);
+ return await db.tx(async () => {
+ const old = await q.activeKeys.all(id);
const graceUntil = grace ? now() + grace * 1000 : null;
const graceUntil = grace ? now() + grace * 1000 : null;
for (const k of old) {
for (const k of old) {
- if (graceUntil) q.graceKey.run(graceUntil, k.id);
- else q.revokeKey.run(now(), reason, k.id);
+ if (graceUntil) await q.graceKey.run(graceUntil, k.id);
+ else await q.revokeKey.run(now(), reason, k.id);
}
}
- const key = issueKey(id, rotated_by);
- events.enqueue({
+ const key = await issueKey(id, rotated_by);
+ await events.enqueue({
event_type: TYPES.keyRotated,
event_type: TYPES.keyRotated,
actor: rotated_by && /^usr_/.test(rotated_by) ? { type: 'user', id: rotated_by } : { type: 'service', id: rotated_by ? String(rotated_by).replace(/^svc:/, '') : 'openre' },
actor: rotated_by && /^usr_/.test(rotated_by) ? { type: 'user', id: rotated_by } : { type: 'service', id: rotated_by ? String(rotated_by).replace(/^svc:/, '') : 'openre' },
subject: { type: 'stream', id, revision: current.revision },
subject: { type: 'stream', id, revision: current.revision },
@@ -214,18 +214,18 @@ function createDefinitions({ db, config, events, clock }) {
@@ -214,18 +214,18 @@ function createDefinitions({ db, config, events, clock }) {
},
},
});
});
return { key, retired: old.map(k => ({ id: k.id, hint: k.hint, status: graceUntil ? 'grace' : 'revoked' })), grace_until: graceUntil };
return { key, retired: old.map(k => ({ id: k.id, hint: k.hint, status: graceUntil ? 'grace' : 'revoked' })), grace_until: graceUntil };
- })();
+ });
}
}
/** Revoke keys whose grace period is over (coordinator). */
/** Revoke keys whose grace period is over (coordinator). */
- function expireGraceKeys() {
- const rows = q.expiredGrace.all(now());
- for (const r of rows) q.revokeKey.run(now(), 'grace_ended', r.id);
+ async function expireGraceKeys() {
+ const rows = await q.expiredGrace.all(now());
+ for (const r of rows) await q.revokeKey.run(now(), 'grace_ended', r.id);
return rows.length;
return rows.length;
}
}
- function keys(id) {
- return q.keysOf.all(id);
+ async function keys(id) {
+ return await q.keysOf.all(id);
}
}
/**
/**
@@ -233,24 +233,24 @@ function createDefinitions({ db, config, events, clock }) {
@@ -233,24 +233,24 @@ function createDefinitions({ db, config, events, clock }) {
* answered inside node-media-server's prePublish handler. Only OpenRe's own hashed keys count:
* answered inside node-media-server's prePublish handler. Only OpenRe's own hashed keys count:
* this never looks at any other service's database.
* this never looks at any other service's database.
*/
*/
- function resolveIngestKey(plain, protocol) {
+ async function resolveIngestKey(plain, protocol) {
if (!isIngestKeyShape(plain)) return { error: 'malformed_key' };
if (!isIngestKeyShape(plain)) return { error: 'malformed_key' };
- const key = q.keyByHash.get(hashIngestKey(plain));
+ const key = await q.keyByHash.get(hashIngestKey(plain));
if (!key) return { error: 'unknown_key' };
if (!key) return { error: 'unknown_key' };
if (key.status === 'revoked') return { error: 'revoked_key' };
if (key.status === 'revoked') return { error: 'revoked_key' };
if (key.status === 'grace' && !(key.grace_until > now())) return { error: 'revoked_key' };
if (key.status === 'grace' && !(key.grace_until > now())) return { error: 'revoked_key' };
- const definition = row(key.definition_id);
+ const definition = await row(key.definition_id);
if (!definition || definition.state !== 'active') return { error: 'stream_disabled' };
if (!definition || definition.state !== 'active') return { error: 'stream_disabled' };
if (!definition.protocols.includes(protocol)) return { error: 'protocol_not_allowed' };
if (!definition.protocols.includes(protocol)) return { error: 'protocol_not_allowed' };
- q.touchKey.run(now(), key.id);
+ await q.touchKey.run(now(), key.id);
return { definition, key: { id: key.id, hint: key.hint, status: key.status } };
return { definition, key: { id: key.id, hint: key.hint, status: key.status } };
}
}
/** What an encoder needs: server URL and which key (by hint) is current. Never the key. */
/** What an encoder needs: server URL and which key (by hint) is current. Never the key. */
- function ingestEndpoints(definition) {
+ async function ingestEndpoints(definition) {
const r = config.rtmp;
const r = config.rtmp;
const portSuffix = r.publicPort === 1935 ? '' : `:${r.publicPort}`;
const portSuffix = r.publicPort === 1935 ? '' : `:${r.publicPort}`;
- const active = keys(definition.id).find(k => k.status === 'active');
+ const active = (await keys(definition.id)).find(k => k.status === 'active');
const out = {};
const out = {};
if (definition.protocols.includes('rtmp')) {
if (definition.protocols.includes('rtmp')) {
out.rtmp = {
out.rtmp = {
@@ -2,7 +2,7 @@
@@ -2,7 +2,7 @@
/**
/**
* One entry point for every process: open the database, the event outbox and the store modules.
* One entry point for every process: open the database, the event outbox and the store modules.
*
*
- * const rt = openRuntime({ config }); // rt.db, rt.events, rt.store.{definitions,sessions,...}
+ * const rt = await openRuntime({ config }); // rt.db, rt.events, rt.store.{definitions,sessions,...}
*/
*/
const { openDb } = require('../db');
const { openDb } = require('../db');
const { createEvents } = require('../events');
const { createEvents } = require('../events');
@@ -13,7 +13,25 @@ const { createSessions } = require('./sessions');
@@ -13,7 +13,25 @@ const { createSessions } = require('./sessions');
const { createOutputs } = require('./outputs');
const { createOutputs } = require('./outputs');
const { createRecordings } = require('./recordings');
const { createRecordings } = require('./recordings');
+/**
+ * The handle every store gets: its transactions are SERIALIZABLE. Several OpenRe processes change the same rows (a
+ * publish admitted by ingest while the coordinator fails a lost worker's session), and SQLite's one writer used to
+ * order them; PostgreSQL detects the conflict and the SDK runs the loser again (40001 is retried).
+ */
+function serializable(db) {
+ if (db.serializable) return db;
+ return new Proxy(db, {
+ get(target, key) {
+ if (key === 'serializable') return true;
+ if (key === 'tx') return async (fn, opts = {}) => await target.tx(fn, { isolation: 'serializable', ...opts });
+ const v = target[key];
+ return typeof v === 'function' ? v.bind(target) : v;
+ },
+ });
+}
+
function createStore({ db, config, events, clock, box, log = console }) {
function createStore({ db, config, events, clock, box, log = console }) {
+ db = serializable(db);
const definitions = createDefinitions({ db, config, events, clock });
const definitions = createDefinitions({ db, config, events, clock });
const workers = createWorkers({ db, config, clock });
const workers = createWorkers({ db, config, clock });
const sessions = createSessions({ db, config, events, clock, definitions, workers });
const sessions = createSessions({ db, config, events, clock, definitions, workers });
@@ -22,12 +40,12 @@ function createStore({ db, config, events, clock, box, log = console }) {
@@ -22,12 +40,12 @@ function createStore({ db, config, events, clock, box, log = console }) {
return { definitions, workers, sessions, outputs, recordings };
return { definitions, workers, sessions, outputs, recordings };
}
}
-function openRuntime({ config, clock = { now: () => Date.now() }, fetchImpl = globalThis.fetch, log = console, db } = {}) {
- db = db || openDb(config.dbPath);
+async function openRuntime({ config, clock = { now: () => Date.now() }, fetchImpl = globalThis.fetch, log = console, db } = {}) {
+ db = serializable(db || await openDb(config, { log }));
const events = createEvents({ config, db, fetchImpl, now: () => clock.now(), log });
const events = createEvents({ config, db, fetchImpl, now: () => clock.now(), log });
const box = createBox({ key: config.secretsKey, previous: config.secretsKeyPrevious });
const box = createBox({ key: config.secretsKey, previous: config.secretsKeyPrevious });
const store = createStore({ db, config, events, clock, box, log });
const store = createStore({ db, config, events, clock, box, log });
return { db, events, box, store, clock, config };
return { db, events, box, store, clock, config };
}
}
-module.exports = { openRuntime, createStore };
+module.exports = { openRuntime, createStore, serializable };
@@ -40,7 +40,7 @@ function createOutputs({ db, config, events, clock, box, definitions, sessions }
@@ -40,7 +40,7 @@ function createOutputs({ db, config, events, clock, box, definitions, sessions }
WHERE state IN ${OPEN} AND (desired = 'stop' AND (worker_id IS NULL OR worker_id IN (SELECT id FROM workers WHERE state IN ('stopped', 'lost'))))`),
WHERE state IN ${OPEN} AND (desired = 'stop' AND (worker_id IS NULL OR worker_id IN (SELECT id FROM workers WHERE state IN ('stopped', 'lost'))))`),
insertLog: db.prepare('INSERT INTO output_logs (output_id, destination_id, level, message, at) VALUES (?, ?, ?, ?, ?)'),
insertLog: db.prepare('INSERT INTO output_logs (output_id, destination_id, level, message, at) VALUES (?, ?, ?, ?, ?)'),
pruneLogs: db.prepare(`DELETE FROM output_logs WHERE id IN (SELECT id FROM output_logs WHERE destination_id = ? AND
pruneLogs: db.prepare(`DELETE FROM output_logs WHERE id IN (SELECT id FROM output_logs WHERE destination_id = ? AND
- (output_id IS ? OR output_id = ?) ORDER BY id DESC LIMIT -1 OFFSET ?)`),
+ (output_id IS NOT DISTINCT FROM ? OR output_id = ?) ORDER BY id DESC LIMIT ALL OFFSET ?)`),
logsOfOutput: db.prepare('SELECT level, message, at FROM output_logs WHERE output_id = ? ORDER BY id DESC LIMIT ?'),
logsOfOutput: db.prepare('SELECT level, message, at FROM output_logs WHERE output_id = ? ORDER BY id DESC LIMIT ?'),
logsOfDest: db.prepare('SELECT output_id, level, message, at FROM output_logs WHERE destination_id = ? ORDER BY id DESC LIMIT ?'),
logsOfDest: db.prepare('SELECT output_id, level, message, at FROM output_logs WHERE destination_id = ? ORDER BY id DESC LIMIT ?'),
};
};
@@ -126,21 +126,21 @@ function createOutputs({ db, config, events, clock, box, definitions, sessions }
@@ -126,21 +126,21 @@ function createOutputs({ db, config, events, clock, box, definitions, sessions }
return f;
return f;
}
}
- function createDestination(definitionId, input, { hold_reason = null } = {}) {
- const def = definitions.get(definitionId);
+ async function createDestination(definitionId, input, { hold_reason = null } = {}) {
+ const def = await definitions.get(definitionId);
if (!def || def.state === 'archived') throw new StoreError(404, 'openre.stream_not_found', 'no such stream definition');
if (!def || def.state === 'archived') throw new StoreError(404, 'openre.stream_not_found', 'no such stream definition');
- if (q.countOf.get(definitionId).n >= config.outputs.maxPerStream) throw new StoreError(409, 'openre.too_many_destinations', `at most ${config.outputs.maxPerStream} destinations per stream`);
+ if ((await q.countOf.get(definitionId)).n >= config.outputs.maxPerStream) throw new StoreError(409, 'openre.too_many_destinations', `at most ${config.outputs.maxPerStream} destinations per stream`);
const f = destFields(input, { creating: true });
const f = destFields(input, { creating: true });
const id = newId('destination', now());
const id = newId('destination', now());
const row = { id, definition_id: definitionId, enabled: 1, auto_start: 1, quality_preset: 'auto', ...f, hold_reason, created_at: now(), updated_at: now() };
const row = { id, definition_id: definitionId, enabled: 1, auto_start: 1, quality_preset: 'auto', ...f, hold_reason, created_at: now(), updated_at: now() };
if (hold_reason) row.enabled = 0;
if (hold_reason) row.enabled = 0;
const cols = Object.keys(row);
const cols = Object.keys(row);
- db.prepare(`INSERT INTO destinations (${cols.join(', ')}) VALUES (${cols.map(c => `@${c}`).join(', ')})`).run(row);
- return publicDest(q.getDest.get(id));
+ await db.prepare(`INSERT INTO destinations (${cols.join(', ')}) VALUES (${cols.map(c => `@${c}`).join(', ')})`).run(row);
+ return publicDest(await q.getDest.get(id));
}
}
- function updateDestination(id, input) {
- const d = q.getDest.get(id);
+ async function updateDestination(id, input) {
+ const d = await q.getDest.get(id);
if (!d) throw new StoreError(404, 'openre.destination_not_found', 'no such destination');
if (!d) throw new StoreError(404, 'openre.destination_not_found', 'no such destination');
const f = destFields(input, { creating: false });
const f = destFields(input, { creating: false });
// Re-enabling a held destination is an explicit owner decision: the hold is lifted only
// Re-enabling a held destination is an explicit owner decision: the hold is lifted only
@@ -153,42 +153,42 @@ function createOutputs({ db, config, events, clock, box, definitions, sessions }
@@ -153,42 +153,42 @@ function createOutputs({ db, config, events, clock, box, definitions, sessions }
if (!Object.keys(f).length) return publicDest(d);
if (!Object.keys(f).length) return publicDest(d);
f.updated_at = now();
f.updated_at = now();
const cols = Object.keys(f);
const cols = Object.keys(f);
- db.prepare(`UPDATE destinations SET ${cols.map(c => `${c} = @${c}`).join(', ')} WHERE id = @id`).run({ ...f, id });
- return publicDest(q.getDest.get(id));
+ await db.prepare(`UPDATE destinations SET ${cols.map(c => `${c} = @${c}`).join(', ')} WHERE id = @id`).run({ ...f, id });
+ return publicDest(await q.getDest.get(id));
}
}
- function deleteDestination(id) {
- const d = q.getDest.get(id);
+ async function deleteDestination(id) {
+ const d = await q.getDest.get(id);
if (!d) throw new StoreError(404, 'openre.destination_not_found', 'no such destination');
if (!d) throw new StoreError(404, 'openre.destination_not_found', 'no such destination');
- db.transaction(() => {
- db.prepare(`UPDATE outputs SET desired = 'stop', updated_at = ? WHERE destination_id = ? AND state IN ${OPEN}`).run(now(), id);
- const running = db.prepare(`SELECT 1 FROM outputs WHERE destination_id = ? AND state IN ${OPEN} AND worker_id IS NOT NULL`).get(id);
+ await db.tx(async () => {
+ await db.prepare(`UPDATE outputs SET desired = 'stop', updated_at = ? WHERE destination_id = ? AND state IN ${OPEN}`).run(now(), id);
+ const running = await db.prepare(`SELECT 1 FROM outputs WHERE destination_id = ? AND state IN ${OPEN} AND worker_id IS NOT NULL`).get(id);
if (running) throw new StoreError(409, 'openre.destination_running', 'the destination is running; it has been asked to stop, delete it again in a few seconds');
if (running) throw new StoreError(409, 'openre.destination_running', 'the destination is running; it has been asked to stop, delete it again in a few seconds');
- q.deleteDest.run(id);
- })();
+ await q.deleteDest.run(id);
+ });
return true;
return true;
}
}
/** Destination row with its secrets opened, for the restream worker only. */
/** Destination row with its secrets opened, for the restream worker only. */
- function destinationForWorker(id) {
- const d = q.getDest.get(id);
+ async function destinationForWorker(id) {
+ const d = await q.getDest.get(id);
if (!d) return null;
if (!d) return null;
return { ...d, stream_key: box.open(d.stream_key_enc), srt_passphrase: box.open(d.srt_passphrase_enc) };
return { ...d, stream_key: box.open(d.stream_key_enc), srt_passphrase: box.open(d.srt_passphrase_enc) };
}
}
- function markDestinationFailure(id, error) {
- const d = q.getDest.get(id);
+ async function markDestinationFailure(id, error) {
+ const d = await q.getDest.get(id);
if (!d) return null;
if (!d) return null;
const n = (d.consecutive_failures || 0) + 1;
const n = (d.consecutive_failures || 0) + 1;
// Same escalation as Live: 15 min, 1 h, 6 h, then 24 h.
// Same escalation as Live: 15 min, 1 h, 6 h, then 24 h.
const mins = n <= 1 ? 15 : n === 2 ? 60 : n === 3 ? 360 : 1440;
const mins = n <= 1 ? 15 : n === 2 ? 60 : n === 3 ? 360 : 1440;
- db.prepare('UPDATE destinations SET consecutive_failures = ?, last_error = ?, last_failed_at = ?, cooldown_until = ?, updated_at = ? WHERE id = ?')
+ await db.prepare('UPDATE destinations SET consecutive_failures = ?, last_error = ?, last_failed_at = ?, cooldown_until = ?, updated_at = ? WHERE id = ?')
.run(n, String(error || 'restream failed to go live').slice(0, 300), now(), now() + mins * 60000, now(), id);
.run(n, String(error || 'restream failed to go live').slice(0, 300), now(), now() + mins * 60000, now(), id);
return { failures: n, cooldownMinutes: mins };
return { failures: n, cooldownMinutes: mins };
}
}
- function clearDestinationCooldown(id) {
- db.prepare('UPDATE destinations SET consecutive_failures = 0, cooldown_until = NULL, last_error = NULL WHERE id = ? AND (consecutive_failures > 0 OR cooldown_until IS NOT NULL OR last_error IS NOT NULL)').run(id);
+ async function clearDestinationCooldown(id) {
+ await db.prepare('UPDATE destinations SET consecutive_failures = 0, cooldown_until = NULL, last_error = NULL WHERE id = ? AND (consecutive_failures > 0 OR cooldown_until IS NOT NULL OR last_error IS NOT NULL)').run(id);
}
}
// ── Outputs ───────────────────────────────────────────────
// ── Outputs ───────────────────────────────────────────────
@@ -217,13 +217,13 @@ function createOutputs({ db, config, events, clock, box, definitions, sessions }
@@ -217,13 +217,13 @@ function createOutputs({ db, config, events, clock, box, definitions, sessions }
}
}
/** Coordinator: every live session gets an output per enabled, auto-start, usable destination. */
/** Coordinator: every live session gets an output per enabled, auto-start, usable destination. */
- function ensureAutoOutputs() {
- const rows = db.prepare(`SELECT s.id AS session_id, d.id AS destination_id FROM ingest_sessions s
+ async function ensureAutoOutputs() {
+ const rows = await db.prepare(`SELECT s.id AS session_id, d.id AS destination_id FROM ingest_sessions s
JOIN destinations d ON d.definition_id = s.definition_id
JOIN destinations d ON d.definition_id = s.definition_id
LEFT JOIN outputs o ON o.session_id = s.id AND o.destination_id = d.id
LEFT JOIN outputs o ON o.session_id = s.id AND o.destination_id = d.id
WHERE s.state = 'live' AND d.enabled = 1 AND d.auto_start = 1 AND d.hold_reason IS NULL
WHERE s.state = 'live' AND d.enabled = 1 AND d.auto_start = 1 AND d.hold_reason IS NULL
AND d.stream_key_enc IS NOT NULL AND (d.cooldown_until IS NULL OR d.cooldown_until <= ?) AND o.id IS NULL`).all(now());
AND d.stream_key_enc IS NOT NULL AND (d.cooldown_until IS NULL OR d.cooldown_until <= ?) AND o.id IS NULL`).all(now());
- for (const r of rows) q.insertOut.run(newId('output', now()), r.session_id, r.destination_id, now(), now());
+ for (const r of rows) await q.insertOut.run(newId('output', now()), r.session_id, r.destination_id, now(), now());
return rows.length;
return rows.length;
}
}
@@ -231,44 +231,44 @@ function createOutputs({ db, config, events, clock, box, definitions, sessions }
@@ -231,44 +231,44 @@ function createOutputs({ db, config, events, clock, box, definitions, sessions }
* Manual start (the owner pressed Start): creates or re-arms the output for the destination's
* Manual start (the owner pressed Start): creates or re-arms the output for the destination's
* current live session. A manual start ignores a failure cooldown, like Live's.
* current live session. A manual start ignores a failure cooldown, like Live's.
*/
*/
- function startDestination(destinationId) {
- const d = q.getDest.get(destinationId);
+ async function startDestination(destinationId) {
+ const d = await q.getDest.get(destinationId);
if (!d) throw new StoreError(404, 'openre.destination_not_found', 'no such destination');
if (!d) throw new StoreError(404, 'openre.destination_not_found', 'no such destination');
if (d.hold_reason) throw new StoreError(409, 'openre.destination_held', `held: ${d.hold_reason}`);
if (d.hold_reason) throw new StoreError(409, 'openre.destination_held', `held: ${d.hold_reason}`);
if (!d.enabled) throw new StoreError(409, 'openre.destination_disabled', 'enable the destination first');
if (!d.enabled) throw new StoreError(409, 'openre.destination_disabled', 'enable the destination first');
if (!d.stream_key_enc) throw new StoreError(409, 'openre.destination_no_key', 'the destination has no stream key');
if (!d.stream_key_enc) throw new StoreError(409, 'openre.destination_no_key', 'the destination has no stream key');
- const session = sessions.list({ definition_id: d.definition_id, state: 'live', limit: 1 })[0];
+ const session = (await sessions.list({ definition_id: d.definition_id, state: 'live', limit: 1 }))[0];
if (!session) throw new StoreError(409, 'openre.not_live', 'the stream is not live');
if (!session) throw new StoreError(409, 'openre.not_live', 'the stream is not live');
- return db.transaction(() => {
- clearDestinationCooldown(d.id);
- const existing = q.outBySessionDest.get(session.id, d.id);
+ return await db.tx(async () => {
+ await clearDestinationCooldown(d.id);
+ const existing = await q.outBySessionDest.get(session.id, d.id);
if (!existing) {
if (!existing) {
const id = newId('output', now());
const id = newId('output', now());
- q.insertOut.run(id, session.id, d.id, now(), now());
- return publicOutput(q.getOut.get(id));
+ await q.insertOut.run(id, session.id, d.id, now(), now());
+ return publicOutput(await q.getOut.get(id));
}
}
if (['pending', 'starting', 'live', 'error'].includes(existing.state) && existing.desired === 'run') return publicOutput(existing);
if (['pending', 'starting', 'live', 'error'].includes(existing.state) && existing.desired === 'run') return publicOutput(existing);
if (['pending', 'starting', 'live', 'error'].includes(existing.state)) {
if (['pending', 'starting', 'live', 'error'].includes(existing.state)) {
- db.prepare("UPDATE outputs SET desired = 'run', updated_at = ? WHERE id = ?").run(now(), existing.id);
+ await db.prepare("UPDATE outputs SET desired = 'run', updated_at = ? WHERE id = ?").run(now(), existing.id);
} else {
} else {
- db.prepare(`UPDATE outputs SET desired = 'run', state = 'pending', worker_id = NULL, worker_generation = NULL,
+ await db.prepare(`UPDATE outputs SET desired = 'run', state = 'pending', worker_id = NULL, worker_generation = NULL,
restart_attempts = 0, next_restart_at = NULL, last_error = NULL, ended_at = NULL, updated_at = ? WHERE id = ?`).run(now(), existing.id);
restart_attempts = 0, next_restart_at = NULL, last_error = NULL, ended_at = NULL, updated_at = ? WHERE id = ?`).run(now(), existing.id);
}
}
- return publicOutput(q.getOut.get(existing.id));
- })();
+ return publicOutput(await q.getOut.get(existing.id));
+ });
}
}
- function stopDestination(destinationId) {
- const n = db.prepare(`UPDATE outputs SET desired = 'stop', updated_at = ? WHERE destination_id = ? AND desired = 'run' AND state IN ${OPEN}`).run(now(), destinationId).changes;
+ async function stopDestination(destinationId) {
+ const n = (await db.prepare(`UPDATE outputs SET desired = 'stop', updated_at = ? WHERE destination_id = ? AND desired = 'run' AND state IN ${OPEN}`).run(now(), destinationId)).changes;
// Pending outputs nobody has picked up can stop right here.
// Pending outputs nobody has picked up can stop right here.
- db.prepare("UPDATE outputs SET state = 'stopped', ended_at = ?, updated_at = ? WHERE destination_id = ? AND desired = 'stop' AND state = 'pending' AND worker_id IS NULL").run(now(), now(), destinationId);
+ await db.prepare("UPDATE outputs SET state = 'stopped', ended_at = ?, updated_at = ? WHERE destination_id = ? AND desired = 'stop' AND state = 'pending' AND worker_id IS NULL").run(now(), now(), destinationId);
return n;
return n;
}
}
/** Coordinator: hand pending outputs to the newest ready restream generation. */
/** Coordinator: hand pending outputs to the newest ready restream generation. */
- function assignPending(worker) {
+ async function assignPending(worker) {
let n = 0;
let n = 0;
- for (const o of q.pendingUnassigned.all()) n += q.assign.run(worker.id, worker.generation, now(), o.id).changes;
+ for (const o of await q.pendingUnassigned.all()) n += (await q.assign.run(worker.id, worker.generation, now(), o.id)).changes;
return n;
return n;
}
}
@@ -278,9 +278,9 @@ function createOutputs({ db, config, events, clock, box, definitions, sessions }
@@ -278,9 +278,9 @@ function createOutputs({ db, config, events, clock, box, definitions, sessions }
* is still assigned to that worker: a worker the coordinator has given up (lost) or that handed
* is still assigned to that worker: a worker the coordinator has given up (lost) or that handed
* the output over must not overwrite what the new owner does.
* the output over must not overwrite what the new owner does.
*/
*/
- function report(outputId, change, { workerId } = {}) {
- return db.transaction(() => {
- const o = q.getOut.get(outputId);
+ async function report(outputId, change, { workerId } = {}) {
+ return await db.tx(async () => {
+ const o = await q.getOut.get(outputId);
if (!o) return null;
if (!o) return null;
if (workerId && o.worker_id !== workerId) return null;
if (workerId && o.worker_id !== workerId) return null;
const set = { updated_at: now() };
const set = { updated_at: now() };
@@ -290,13 +290,13 @@ function createOutputs({ db, config, events, clock, box, definitions, sessions }
@@ -290,13 +290,13 @@ function createOutputs({ db, config, events, clock, box, definitions, sessions }
if (change.progress !== undefined) set.progress = change.progress ? JSON.stringify(change.progress) : null;
if (change.progress !== undefined) set.progress = change.progress ? JSON.stringify(change.progress) : null;
if (change.state === 'live') set.ever_live = 1;
if (change.state === 'live') set.ever_live = 1;
const cols = Object.keys(set);
const cols = Object.keys(set);
- db.prepare(`UPDATE outputs SET ${cols.map(c => `${c} = @${c}`).join(', ')} WHERE id = @id`).run({ ...set, id: outputId });
- const after = q.getOut.get(outputId);
+ await db.prepare(`UPDATE outputs SET ${cols.map(c => `${c} = @${c}`).join(', ')} WHERE id = @id`).run({ ...set, id: outputId });
+ const after = await q.getOut.get(outputId);
if (change.state && change.state !== o.state && (change.state === 'live' || change.state === 'failed')) {
if (change.state && change.state !== o.state && (change.state === 'live' || change.state === 'failed')) {
- const dest = q.getDest.get(o.destination_id);
- const session = sessions.get(o.session_id);
+ const dest = await q.getDest.get(o.destination_id);
+ const session = await sessions.get(o.session_id);
if (dest && session) {
if (dest && session) {
- events.enqueue({
+ await events.enqueue({
event_type: change.state === 'live' ? TYPES.outputHealthy : TYPES.outputFailed,
event_type: change.state === 'live' ? TYPES.outputHealthy : TYPES.outputFailed,
actor: { type: 'service', id: 'openre' },
actor: { type: 'service', id: 'openre' },
subject: { type: 'output', id: o.id, revision: after.restart_attempts + 1 },
subject: { type: 'output', id: o.id, revision: after.restart_attempts + 1 },
@@ -315,43 +315,43 @@ function createOutputs({ db, config, events, clock, box, definitions, sessions }
@@ -315,43 +315,43 @@ function createOutputs({ db, config, events, clock, box, definitions, sessions }
}
}
}
}
return after;
return after;
- })();
+ });
}
}
- function log(outputId, destinationId, level, message) {
+ async function log(outputId, destinationId, level, message) {
const text = String(message || '').slice(0, 500);
const text = String(message || '').slice(0, 500);
- q.insertLog.run(outputId || null, destinationId, level, text, now());
- q.pruneLogs.run(destinationId, outputId || null, outputId || null, config.outputs.logsPerOutput);
+ await q.insertLog.run(outputId || null, destinationId, level, text, now());
+ await q.pruneLogs.run(destinationId, outputId || null, outputId || null, config.outputs.logsPerOutput);
}
}
return {
return {
PLATFORMS, QUALITY_PRESETS, ENCODER_PRESETS,
PLATFORMS, QUALITY_PRESETS, ENCODER_PRESETS,
publicDest,
publicDest,
publicOutput,
publicOutput,
- getDestination: (id) => (isId('destination', id) ? publicDest(q.getDest.get(id)) : null),
- destinationRow: (id) => q.getDest.get(id),
- destinations: (definitionId) => q.destsOf.all(definitionId).map(publicDest),
+ getDestination: async (id) => (isId('destination', id) ? publicDest(await q.getDest.get(id)) : null),
+ destinationRow: async (id) => await q.getDest.get(id),
+ destinations: async (definitionId) => (await q.destsOf.all(definitionId)).map(publicDest),
createDestination,
createDestination,
updateDestination,
updateDestination,
deleteDestination,
deleteDestination,
destinationForWorker,
destinationForWorker,
markDestinationFailure,
markDestinationFailure,
clearDestinationCooldown,
clearDestinationCooldown,
- getOutput: (id) => (isId('output', id) ? publicOutput(q.getOut.get(id)) : null),
- outputRow: (id) => q.getOut.get(id),
- outputsOfSession: (sessionId) => q.outsOfSession.all(sessionId).map(publicOutput),
+ getOutput: async (id) => (isId('output', id) ? publicOutput(await q.getOut.get(id)) : null),
+ outputRow: async (id) => await q.getOut.get(id),
+ outputsOfSession: async (sessionId) => (await q.outsOfSession.all(sessionId)).map(publicOutput),
ensureAutoOutputs,
ensureAutoOutputs,
startDestination,
startDestination,
stopDestination,
stopDestination,
assignPending,
assignPending,
- forWorker: (workerId) => q.forWorker.all(workerId),
- release: (workerId) => q.release.run(now(), workerId).changes,
- unassign: (outputId) => q.unassign.run(now(), outputId).changes,
- stopOrphans: () => q.stopOrphans.run(now(), now()).changes,
+ forWorker: async (workerId) => await q.forWorker.all(workerId),
+ release: async (workerId) => (await q.release.run(now(), workerId)).changes,
+ unassign: async (outputId) => (await q.unassign.run(now(), outputId)).changes,
+ stopOrphans: async () => (await q.stopOrphans.run(now(), now())).changes,
report,
report,
log,
log,
- logsOfOutput: (id, limit = 100) => q.logsOfOutput.all(id, limit),
- logsOfDestination: (id, limit = 100) => q.logsOfDest.all(id, limit),
+ logsOfOutput: async (id, limit = 100) => await q.logsOfOutput.all(id, limit),
+ logsOfDestination: async (id, limit = 100) => await q.logsOfDest.all(id, limit),
};
};
}
}
@@ -38,23 +38,23 @@ function createRecordings({ db, config, events, clock, definitions, sessions, lo
@@ -38,23 +38,23 @@ function createRecordings({ db, config, events, clock, definitions, sessions, lo
set: (cols) => db.prepare(`UPDATE recordings SET ${cols.map(c => `${c} = @${c}`).join(', ')}, updated_at = @now WHERE id = @id`),
set: (cols) => db.prepare(`UPDATE recordings SET ${cols.map(c => `${c} = @${c}`).join(', ')}, updated_at = @now WHERE id = @id`),
};
};
- function update(id, fields) {
- q.set(Object.keys(fields)).run({ ...fields, id, now: now() });
- return q.get.get(id);
+ async function update(id, fields) {
+ await q.set(Object.keys(fields)).run({ ...fields, id, now: now() });
+ return await q.get.get(id);
}
}
- function ensureRequests() {
- const rows = q.missing.all(now() - config.media.startDelayMs);
- for (const r of rows) q.insert.run(newId('recording', now()), r.session_id, r.mode, config.media.appId, now(), now());
+ async function ensureRequests() {
+ const rows = await q.missing.all(now() - config.media.startDelayMs);
+ for (const r of rows) await q.insert.run(newId('recording', now()), r.session_id, r.mode, config.media.appId, now(), now());
return rows.length;
return rows.length;
}
}
- function retryLater(rec, err, { disk = false } = {}) {
+ async function retryLater(rec, err, { disk = false } = {}) {
const attempts = rec.attempts + 1;
const attempts = rec.attempts + 1;
const limit = disk ? DISK_MAX_ATTEMPTS : MAX_ATTEMPTS;
const limit = disk ? DISK_MAX_ATTEMPTS : MAX_ATTEMPTS;
- if (attempts >= limit) return update(rec.id, { attempts, state: 'failed', last_error: String(err.message || err).slice(0, 300) });
+ if (attempts >= limit) return await update(rec.id, { attempts, state: 'failed', last_error: String(err.message || err).slice(0, 300) });
const wait = disk ? DISK_RETRY_MS : BACKOFF_MS[Math.min(attempts - 1, BACKOFF_MS.length - 1)];
const wait = disk ? DISK_RETRY_MS : BACKOFF_MS[Math.min(attempts - 1, BACKOFF_MS.length - 1)];
- return update(rec.id, { attempts, next_attempt_at: now() + wait, last_error: String(err.message || err).slice(0, 300) });
+ return await update(rec.id, { attempts, next_attempt_at: now() + wait, last_error: String(err.message || err).slice(0, 300) });
}
}
function refOf(definition, service, type) {
function refOf(definition, service, type) {
@@ -63,15 +63,15 @@ function createRecordings({ db, config, events, clock, definitions, sessions, lo
@@ -63,15 +63,15 @@ function createRecordings({ db, config, events, clock, definitions, sessions, lo
}
}
async function step(rec, media) {
async function step(rec, media) {
- const session = sessions.get(rec.session_id);
- if (!session) return update(rec.id, { state: 'failed', last_error: 'session missing' });
- const definition = definitions.row(session.definition_id);
+ const session = await sessions.get(rec.session_id);
+ if (!session) return await update(rec.id, { state: 'failed', last_error: 'session missing' });
+ const definition = await definitions.row(session.definition_id);
const ended = isTerminal(session.state) || session.state === 'ending';
const ended = isTerminal(session.state) || session.state === 'ending';
if (rec.state === 'pending') {
if (rec.state === 'pending') {
- if (ended) return update(rec.id, { state: 'cancelled' });
- const pb = sessions.playback(session);
- if (!pb || !pb.rtmp) return retryLater(rec, new Error('no loopback play URL for this session yet'));
+ if (ended) return await update(rec.id, { state: 'cancelled' });
+ const pb = await sessions.playback(session);
+ if (!pb || !pb.rtmp) return await retryLater(rec, new Error('no loopback play URL for this session yet'));
let vodId = rec.media_vod_id;
let vodId = rec.media_vod_id;
try {
try {
if (!vodId) {
if (!vodId) {
@@ -88,7 +88,7 @@ function createRecordings({ db, config, events, clock, definitions, sessions, lo
@@ -88,7 +88,7 @@ function createRecordings({ db, config, events, clock, definitions, sessions, lo
meta: { source: 'openre', openre_session_id: session.id, openre_stream_id: definition.id, protocol: session.protocol, mode: rec.mode },
meta: { source: 'openre', openre_session_id: session.id, openre_stream_id: definition.id, protocol: session.protocol, mode: rec.mode },
});
});
vodId = String(out && out.id);
vodId = String(out && out.id);
- rec = update(rec.id, { media_vod_id: vodId, state: 'requested' });
+ rec = await update(rec.id, { media_vod_id: vodId, state: 'requested' });
}
}
await media.ingestRtmp(vodId, pb.rtmp.internal_url);
await media.ingestRtmp(vodId, pb.rtmp.internal_url);
} catch (err) {
} catch (err) {
@@ -96,12 +96,12 @@ function createRecordings({ db, config, events, clock, definitions, sessions, lo
@@ -96,12 +96,12 @@ function createRecordings({ db, config, events, clock, definitions, sessions, lo
// Media may have made the VOD row before the ingest was refused (disk low): drop the
// Media may have made the VOD row before the ingest was refused (disk low): drop the
// empty shell so it never shows as a 0:00 VOD, and start over next time.
// empty shell so it never shows as a 0:00 VOD, and start over next time.
if (vodId) await media.deleteVod(vodId).catch(() => {});
if (vodId) await media.deleteVod(vodId).catch(() => {});
- update(rec.id, { media_vod_id: null, state: 'pending' });
- return retryLater(q.get.get(rec.id), err, { disk: /disk/i.test(String(err.message)) });
+ await update(rec.id, { media_vod_id: null, state: 'pending' });
+ return await retryLater(await q.get.get(rec.id), err, { disk: /disk/i.test(String(err.message)) });
}
}
- return db.transaction(() => {
- const r = update(rec.id, { state: 'recording', attempts: 0, last_error: null, next_attempt_at: 0 });
- events.enqueue({
+ return await db.tx(async () => {
+ const r = await update(rec.id, { state: 'recording', attempts: 0, last_error: null, next_attempt_at: 0 });
+ await events.enqueue({
event_type: TYPES.recordingRequested,
event_type: TYPES.recordingRequested,
actor: { type: 'service', id: 'openre' },
actor: { type: 'service', id: 'openre' },
subject: { type: 'recording', id: rec.id, revision: 1 },
subject: { type: 'recording', id: rec.id, revision: 1 },
@@ -118,25 +118,25 @@ function createRecordings({ db, config, events, clock, definitions, sessions, lo
@@ -118,25 +118,25 @@ function createRecordings({ db, config, events, clock, definitions, sessions, lo
},
},
});
});
return r;
return r;
- })();
+ });
}
}
if (rec.state === 'requested') {
if (rec.state === 'requested') {
// A create that succeeded and an ingest that did not: handled above; a requested row
// A create that succeeded and an ingest that did not: handled above; a requested row
// here means the process died in between. Start the ingest again.
// here means the process died in between. Start the ingest again.
- return update(rec.id, { state: 'pending' });
+ return await update(rec.id, { state: 'pending' });
}
}
- if (rec.state === 'recording' && ended) rec = update(rec.id, { state: 'finalizing' });
+ if (rec.state === 'recording' && ended) rec = await update(rec.id, { state: 'finalizing' });
if (rec.state === 'finalizing') {
if (rec.state === 'finalizing') {
try {
try {
await media.finalizeVod(rec.media_vod_id);
await media.finalizeVod(rec.media_vod_id);
if (rec.mode === 'clips') await media.deleteVod(rec.media_vod_id).catch(() => {});
if (rec.mode === 'clips') await media.deleteVod(rec.media_vod_id).catch(() => {});
} catch (err) {
} catch (err) {
// 409 = Media already finalising it (its ffmpeg saw the source end): that is success.
// 409 = Media already finalising it (its ffmpeg saw the source end): that is success.
- if (err.status !== 409) return retryLater(rec, err);
+ if (err.status !== 409) return await retryLater(rec, err);
}
}
- return update(rec.id, { state: 'finalized', last_error: null });
+ return await update(rec.id, { state: 'finalized', last_error: null });
}
}
return rec;
return rec;
}
}
@@ -145,7 +145,7 @@ function createRecordings({ db, config, events, clock, definitions, sessions, lo
@@ -145,7 +145,7 @@ function createRecordings({ db, config, events, clock, definitions, sessions, lo
async function process(media) {
async function process(media) {
if (!media.configured) return 0;
if (!media.configured) return 0;
let n = 0;
let n = 0;
- for (const rec of q.due.all(now())) {
+ for (const rec of await q.due.all(now())) {
try { await step(rec, media); n++; } catch (err) { log.error(`[recording] ${rec.id}: ${err.stack || err}`); }
try { await step(rec, media); n++; } catch (err) { log.error(`[recording] ${rec.id}: ${err.stack || err}`); }
}
}
return n;
return n;
@@ -154,7 +154,7 @@ function createRecordings({ db, config, events, clock, definitions, sessions, lo
@@ -154,7 +154,7 @@ function createRecordings({ db, config, events, clock, definitions, sessions, lo
return {
return {
ensureRequests,
ensureRequests,
process,
process,
- bySession: (sessionId) => q.bySession.get(sessionId) || null,
+ bySession: async (sessionId) => await q.bySession.get(sessionId) || null,
};
};
}
}
@@ -37,8 +37,8 @@ function createSessions({ db, config, events, clock, definitions, workers }) {
@@ -37,8 +37,8 @@ function createSessions({ db, config, events, clock, definitions, workers }) {
};
};
/** The Live channel this definition belongs to (the resolver's answer), or null while unresolved. */
/** The Live channel this definition belongs to (the resolver's answer), or null while unresolved. */
- function liveLineage(definitionId) {
- const row = q.lineage.get(definitionId);
+ async function liveLineage(definitionId) {
+ const row = await q.lineage.get(definitionId);
const r = row ? parseJson(row.resolution, null) : null;
const r = row ? parseJson(row.resolution, null) : null;
if (!r || r.status !== 'resolved' || !r.channel) return null;
if (!r || r.status !== 'resolved' || !r.channel) return null;
return { channel: { service: 'live', id: String(r.channel.id), slug: r.channel.slug, owner_subject: r.channel.owner_subject || null }, rule: r.rule || null };
return { channel: { service: 'live', id: String(r.channel.id), slug: r.channel.slug, owner_subject: r.channel.owner_subject || null }, rule: r.rule || null };
@@ -49,11 +49,11 @@ function createSessions({ db, config, events, clock, definitions, workers }) {
@@ -49,11 +49,11 @@ function createSessions({ db, config, events, clock, definitions, workers }) {
return { ...s, media_info: parseJson(s.media_info, null) };
return { ...s, media_info: parseJson(s.media_info, null) };
}
}
- function get(id) {
- return isId('session', id) ? shape(q.get.get(id)) : null;
+ async function get(id) {
+ return isId('session', id) ? shape(await q.get.get(id)) : null;
}
}
- function list({ definition_id, owner_subject, state, limit = 50, before } = {}) {
+ async function list({ definition_id, owner_subject, state, limit = 50, before } = {}) {
const where = [];
const where = [];
const args = [];
const args = [];
if (definition_id) { where.push('s.definition_id = ?'); args.push(definition_id); }
if (definition_id) { where.push('s.definition_id = ?'); args.push(definition_id); }
@@ -61,13 +61,13 @@ function createSessions({ db, config, events, clock, definitions, workers }) {
@@ -61,13 +61,13 @@ function createSessions({ db, config, events, clock, definitions, workers }) {
if (state === 'open') where.push(`s.state IN ${OPEN}`);
if (state === 'open') where.push(`s.state IN ${OPEN}`);
else if (state) { where.push('s.state = ?'); args.push(state); }
else if (state) { where.push('s.state = ?'); args.push(state); }
if (before) { where.push('s.created_at < ?'); args.push(Number(before)); }
if (before) { where.push('s.created_at < ?'); args.push(Number(before)); }
- return db.prepare(`SELECT s.* FROM ingest_sessions s JOIN stream_definitions d ON d.id = s.definition_id
+ return (await db.prepare(`SELECT s.* FROM ingest_sessions s JOIN stream_definitions d ON d.id = s.definition_id
${where.length ? `WHERE ${where.join(' AND ')}` : ''} ORDER BY s.created_at DESC LIMIT ?`)
${where.length ? `WHERE ${where.join(' AND ')}` : ''} ORDER BY s.created_at DESC LIMIT ?`)
- .all(...args, Math.min(Math.max(Number(limit) || 50, 1), 200)).map(shape);
+ .all(...args, Math.min(Math.max(Number(limit) || 50, 1), 200))).map(shape);
}
}
- function envelopeFor(type, session, definition, extra = {}) {
- const worker = session.worker_id ? workers.get(session.worker_id) : null;
+ async function envelopeFor(type, session, definition, extra = {}) {
+ const worker = session.worker_id ? await workers.get(session.worker_id) : null;
const payload = {
const payload = {
session_id: session.id,
session_id: session.id,
stream_id: definition.id,
stream_id: definition.id,
@@ -79,7 +79,7 @@ function createSessions({ db, config, events, clock, definitions, workers }) {
@@ -79,7 +79,7 @@ function createSessions({ db, config, events, clock, definitions, workers }) {
worker: worker ? { id: worker.id, kind: worker.kind, generation: worker.generation } : null,
worker: worker ? { id: worker.id, kind: worker.kind, generation: worker.generation } : null,
external_refs: definition.external_refs,
external_refs: definition.external_refs,
mirror_to_live: definition.mirror_to_live,
mirror_to_live: definition.mirror_to_live,
- lineage: liveLineage(definition.id),
+ lineage: await liveLineage(definition.id),
...extra,
...extra,
};
};
return {
return {
@@ -93,31 +93,31 @@ function createSessions({ db, config, events, clock, definitions, workers }) {
@@ -93,31 +93,31 @@ function createSessions({ db, config, events, clock, definitions, workers }) {
}
}
/**
/**
- * Admit a publish on `worker` (synchronous; called inside the RTMP handshake).
+ * Admit a publish on `worker` (called inside the RTMP handshake).
* One open session per definition: a second publisher with a valid key is refused while the
* One open session per definition: a second publisher with a valid key is refused while the
* first is alive, exactly like Live's in-process ingest. A leftover open session whose lease
* first is alive, exactly like Live's in-process ingest. A leftover open session whose lease
* has expired (its worker died) is failed first, so a crashed worker never locks a stream out.
* has expired (its worker died) is failed first, so a crashed worker never locks a stream out.
*/
*/
- function admit({ definition, key, protocol, worker }) {
- return db.transaction(() => {
- for (const open of q.openForDefinition.all(definition.id)) {
+ async function admit({ definition, key, protocol, worker }) {
+ return await db.tx(async () => {
+ for (const open of await q.openForDefinition.all(definition.id)) {
const leaseOk = open.lease_expires_at && open.lease_expires_at > now();
const leaseOk = open.lease_expires_at && open.lease_expires_at > now();
if (leaseOk) return { error: 'duplicate_publisher', existing: open.id };
if (leaseOk) return { error: 'duplicate_publisher', existing: open.id };
- applyTransition(open, 'failed', { reason: 'lease_expired', actor: `worker:${worker.id}` });
+ await applyTransition(open, 'failed', { reason: 'lease_expired', actor: `worker:${worker.id}` });
}
}
const id = newId('session', now());
const id = newId('session', now());
- q.insert.run({
+ await q.insert.run({
id, definition_id: definition.id, key_id: key ? key.id : null, protocol,
id, definition_id: definition.id, key_id: key ? key.id : null, protocol,
worker_id: worker.id, worker_kind: worker.kind, worker_generation: worker.generation,
worker_id: worker.id, worker_kind: worker.kind, worker_generation: worker.generation,
lease: now() + config.workers.leaseMs, now: now(),
lease: now() + config.workers.leaseMs, now: now(),
});
});
- q.transitionRow.run(id, null, 'starting', 'publish_accepted', `worker:${worker.id}`, now());
- return { session: shape(q.get.get(id)) };
- }).immediate();
+ await q.transitionRow.run(id, null, 'starting', 'publish_accepted', `worker:${worker.id}`, now());
+ return { session: shape(await q.get.get(id)) };
+ });
}
}
// Inside a transaction.
// Inside a transaction.
- function applyTransition(current, to, { reason = null, actor = null, media_info } = {}) {
+ async function applyTransition(current, to, { reason = null, actor = null, media_info } = {}) {
if (!canTransition(current.state, to)) return { ok: false, code: 'invalid_transition', session: shape(current) };
if (!canTransition(current.state, to)) return { ok: false, code: 'invalid_transition', session: shape(current) };
const t = now();
const t = now();
const set = { state: to, updated_at: t, revision: current.revision + 1 };
const set = { state: to, updated_at: t, revision: current.revision + 1 };
@@ -128,21 +128,21 @@ function createSessions({ db, config, events, clock, definitions, workers }) {
@@ -128,21 +128,21 @@ function createSessions({ db, config, events, clock, definitions, workers }) {
if (media_info !== undefined) set.media_info = media_info ? JSON.stringify(media_info) : null;
if (media_info !== undefined) set.media_info = media_info ? JSON.stringify(media_info) : null;
if (isTerminal(to)) set.lease_expires_at = null;
if (isTerminal(to)) set.lease_expires_at = null;
const cols = Object.keys(set);
const cols = Object.keys(set);
- db.prepare(`UPDATE ingest_sessions SET ${cols.map(c => `${c} = @${c}`).join(', ')} WHERE id = @id AND state = @from`)
+ await db.prepare(`UPDATE ingest_sessions SET ${cols.map(c => `${c} = @${c}`).join(', ')} WHERE id = @id AND state = @from`)
.run({ ...set, id: current.id, from: current.state });
.run({ ...set, id: current.id, from: current.state });
- q.transitionRow.run(current.id, current.state, to, reason, actor, t);
- const session = shape(q.get.get(current.id));
- const definition = definitions.row(session.definition_id);
+ await q.transitionRow.run(current.id, current.state, to, reason, actor, t);
+ const session = shape(await q.get.get(current.id));
+ const definition = await definitions.row(session.definition_id);
if (to === 'live') {
if (to === 'live') {
- events.enqueue(envelopeFor(TYPES.sessionStarted, session, definition, { playback: playback(session) }));
+ await events.enqueue(await envelopeFor(TYPES.sessionStarted, session, definition, { playback: await playback(session) }));
} else if (to === 'ended' && session.live_at) {
} else if (to === 'ended' && session.live_at) {
- events.enqueue(envelopeFor(TYPES.sessionEnded, session, definition, {
+ await events.enqueue(await envelopeFor(TYPES.sessionEnded, session, definition, {
ended_at: iso(session.ended_at),
ended_at: iso(session.ended_at),
duration_seconds: Math.round((session.ended_at - session.live_at) / 1000),
duration_seconds: Math.round((session.ended_at - session.live_at) / 1000),
end_reason: session.end_reason,
end_reason: session.end_reason,
}));
}));
} else if (to === 'failed') {
} else if (to === 'failed') {
- events.enqueue(envelopeFor(TYPES.sessionFailed, session, definition, {
+ await events.enqueue(await envelopeFor(TYPES.sessionFailed, session, definition, {
ended_at: iso(session.ended_at),
ended_at: iso(session.ended_at),
was_live: Boolean(session.live_at),
was_live: Boolean(session.live_at),
duration_seconds: session.live_at ? Math.round((session.ended_at - session.live_at) / 1000) : 0,
duration_seconds: session.live_at ? Math.round((session.ended_at - session.live_at) / 1000) : 0,
@@ -151,45 +151,45 @@ function createSessions({ db, config, events, clock, definitions, workers }) {
@@ -151,45 +151,45 @@ function createSessions({ db, config, events, clock, definitions, workers }) {
}
}
if (isTerminal(to) || to === 'ending') {
if (isTerminal(to) || to === 'ending') {
// Outputs follow their session down; they never pull the session with them.
// Outputs follow their session down; they never pull the session with them.
- db.prepare("UPDATE outputs SET desired = 'stop', updated_at = ? WHERE session_id = ? AND desired = 'run'").run(t, session.id);
+ await db.prepare("UPDATE outputs SET desired = 'stop', updated_at = ? WHERE session_id = ? AND desired = 'run'").run(t, session.id);
}
}
return { ok: true, session };
return { ok: true, session };
}
}
/** transition(id, to, { reason, actor, media_info }) -> { ok, session } | { ok: false, code } */
/** transition(id, to, { reason, actor, media_info }) -> { ok, session } | { ok: false, code } */
- function transition(id, to, opts = {}) {
- return db.transaction(() => {
- const current = q.get.get(id);
+ async function transition(id, to, opts = {}) {
+ return await db.tx(async () => {
+ const current = await q.get.get(id);
if (!current) return { ok: false, code: 'not_found' };
if (!current) return { ok: false, code: 'not_found' };
- return applyTransition(current, to, opts);
- }).immediate();
+ return await applyTransition(current, to, opts);
+ });
}
}
/** ending → ended in one call (the common clean stop). */
/** ending → ended in one call (the common clean stop). */
- function finish(id, { reason, actor } = {}) {
- return db.transaction(() => {
- let current = q.get.get(id);
+ async function finish(id, { reason, actor } = {}) {
+ return await db.tx(async () => {
+ let current = await q.get.get(id);
if (!current) return { ok: false, code: 'not_found' };
if (!current) return { ok: false, code: 'not_found' };
if (current.state === 'starting' || current.state === 'live') {
if (current.state === 'starting' || current.state === 'live') {
- const r = applyTransition(current, 'ending', { reason, actor });
+ const r = await applyTransition(current, 'ending', { reason, actor });
if (!r.ok) return r;
if (!r.ok) return r;
- current = q.get.get(id);
+ current = await q.get.get(id);
}
}
- return applyTransition(current, 'ended', { reason, actor });
- }).immediate();
+ return await applyTransition(current, 'ended', { reason, actor });
+ });
}
}
- function requestEnd(id, by) {
- return q.requestEnd.run(String(by || 'unknown'), now(), id).changes > 0;
+ async function requestEnd(id, by) {
+ return (await q.requestEnd.run(String(by || 'unknown'), now(), id)).changes > 0;
}
}
/**
/**
* Where to watch a session. Internal URLs are loopback addresses on the OpenRe host (Live and
* Where to watch a session. Internal URLs are loopback addresses on the OpenRe host (Live and
* Media run there too); public_url is served by openre-api's /play proxy.
* Media run there too); public_url is served by openre-api's /play proxy.
*/
*/
- function playback(session) {
+ async function playback(session) {
if (!session) return null;
if (!session) return null;
- const worker = session.worker_id ? workers.get(session.worker_id) : null;
+ const worker = session.worker_id ? await workers.get(session.worker_id) : null;
const ep = (worker && worker.endpoints) || {};
const ep = (worker && worker.endpoints) || {};
const descriptor = {
const descriptor = {
session_id: session.id,
session_id: session.id,
@@ -224,13 +224,13 @@ function createSessions({ db, config, events, clock, definitions, workers }) {
@@ -224,13 +224,13 @@ function createSessions({ db, config, events, clock, definitions, workers }) {
finish,
finish,
requestEnd,
requestEnd,
playback,
playback,
- setMediaInfo: (id, info) => q.mediaInfo.run(JSON.stringify(info), id).changes > 0,
- transitions: (id) => q.transitions.all(id),
- endRequestsFor: (workerId) => q.endRequests.all(workerId).map(shape),
- ofWorker: (workerId) => q.ofWorker.all(workerId).map(shape),
- expiredLeases: () => q.expiredLeases.all(now()).map(shape),
- live: () => q.live.all().map(shape),
- openFor: (definitionId) => q.openForDefinition.all(definitionId).map(shape),
+ setMediaInfo: async (id, info) => (await q.mediaInfo.run(JSON.stringify(info), id)).changes > 0,
+ transitions: async (id) => await q.transitions.all(id),
+ endRequestsFor: async (workerId) => (await q.endRequests.all(workerId)).map(shape),
+ ofWorker: async (workerId) => (await q.ofWorker.all(workerId)).map(shape),
+ expiredLeases: async () => (await q.expiredLeases.all(now())).map(shape),
+ live: async () => (await q.live.all()).map(shape),
+ openFor: async (definitionId) => (await q.openForDefinition.all(definitionId)).map(shape),
};
};
}
}
@@ -44,38 +44,38 @@ function createWorkers({ db, config, clock }) {
@@ -44,38 +44,38 @@ function createWorkers({ db, config, clock }) {
const shape = (w) => (w ? { ...w, endpoints: parseJson(w.endpoints, {}) } : null);
const shape = (w) => (w ? { ...w, endpoints: parseJson(w.endpoints, {}) } : null);
- function register({ kind, release = config.release, pid = process.pid, host = os.hostname(), endpoints = {} }) {
+ async function register({ kind, release = config.release, pid = process.pid, host = os.hostname(), endpoints = {} }) {
if (!KINDS.includes(kind)) throw new Error(`unknown worker kind ${kind}`);
if (!KINDS.includes(kind)) throw new Error(`unknown worker kind ${kind}`);
const id = newId('worker', now());
const id = newId('worker', now());
- db.transaction(() => {
- const generation = q.maxGen.get(kind).g + 1;
- q.insert.run({ id, kind, generation, release, pid, host, endpoints: JSON.stringify(endpoints), now: now() });
- }).immediate();
- return shape(q.get.get(id));
+ await db.tx(async () => {
+ const generation = (await q.maxGen.get(kind)).g + 1;
+ await q.insert.run({ id, kind, generation, release, pid, host, endpoints: JSON.stringify(endpoints), now: now() });
+ });
+ return shape(await q.get.get(id));
}
}
return {
return {
KINDS,
KINDS,
register,
register,
- get: (id) => shape(q.get.get(id)),
- ready: (id) => q.setReady.run(now(), now(), id).changes > 0,
- setEndpoints: (id, endpoints) => q.setEndpoints.run(JSON.stringify(endpoints), id),
+ get: async (id) => shape(await q.get.get(id)),
+ ready: async (id) => (await q.setReady.run(now(), now(), id)).changes > 0,
+ setEndpoints: async (id, endpoints) => await q.setEndpoints.run(JSON.stringify(endpoints), id),
/** Heartbeat + renew the leases of every session this worker owns, in one transaction. */
/** Heartbeat + renew the leases of every session this worker owns, in one transaction. */
- heartbeat(id) {
- return db.transaction(() => {
- const alive = q.heartbeat.run(now(), id).changes > 0;
- if (alive) q.renewLeases.run(now() + config.workers.leaseMs, id);
- return shape(q.get.get(id));
- })();
+ async heartbeat(id) {
+ return await db.tx(async () => {
+ const alive = (await q.heartbeat.run(now(), id)).changes > 0;
+ if (alive) await q.renewLeases.run(now() + config.workers.leaseMs, id);
+ return shape(await q.get.get(id));
+ });
},
},
- drain: (id, deadline) => q.drain.run(now(), deadline, id).changes > 0,
- stop: (id, reason) => q.stop.run(now(), reason || 'exit', id).changes > 0,
- lose: (id, reason) => q.lose.run(now(), reason || 'lease_expired', id).changes > 0,
- newestReady: (kind) => shape(q.newestReady.get(kind)),
- stale: (before) => q.stale.all(before).map(shape),
- olderReady: (kind, generation) => q.olderReady.all(kind, generation).map(shape),
- alive: () => q.alive.all().map(shape),
- recent: (limit = 50) => q.recent.all(limit).map(shape),
+ drain: async (id, deadline) => (await q.drain.run(now(), deadline, id)).changes > 0,
+ stop: async (id, reason) => (await q.stop.run(now(), reason || 'exit', id)).changes > 0,
+ lose: async (id, reason) => (await q.lose.run(now(), reason || 'lease_expired', id)).changes > 0,
+ newestReady: async (kind) => shape(await q.newestReady.get(kind)),
+ stale: async (before) => (await q.stale.all(before)).map(shape),
+ olderReady: async (kind, generation) => (await q.olderReady.all(kind, generation)).map(shape),
+ alive: async () => (await q.alive.all()).map(shape),
+ recent: async (limit = 50) => (await q.recent.all(limit)).map(shape),
};
};
}
}
@@ -53,8 +53,8 @@ function createUiRouter({ rt, auth }) {
@@ -53,8 +53,8 @@ function createUiRouter({ rt, auth }) {
return false;
return false;
}
}
- function ownDefinition(req, res, id) {
- const d = store.definitions.get(id);
+ async function ownDefinition(req, res, id) {
+ const d = await store.definitions.get(id);
if (!d || d.state === 'archived' || !auth.canAccess(req.caller, d.owner_subject)) {
if (!d || d.state === 'archived' || !auth.canAccess(req.caller, d.owner_subject)) {
page(req, res, { title: 'Not found', body: '<h1>Stream not found</h1><p><a href="/streams">Your streams</a></p>' }, 404);
page(req, res, { title: 'Not found', body: '<h1>Stream not found</h1><p><a href="/streams">Your streams</a></p>' }, 404);
return null;
return null;
@@ -62,9 +62,9 @@ function createUiRouter({ rt, auth }) {
@@ -62,9 +62,9 @@ function createUiRouter({ rt, auth }) {
return d;
return d;
}
}
- function ownDestination(req, res, id) {
- const dest = store.outputs.destinationRow(id);
- const d = dest ? ownDefinition(req, res, dest.definition_id) : null;
+ async function ownDestination(req, res, id) {
+ const dest = await store.outputs.destinationRow(id);
+ const d = dest ? await ownDefinition(req, res, dest.definition_id) : null;
if (!dest && !res.headersSent) page(req, res, { title: 'Not found', body: '<h1>Destination not found</h1>' }, 404);
if (!dest && !res.headersSent) page(req, res, { title: 'Not found', body: '<h1>Destination not found</h1>' }, 404);
return d ? { dest, definition: d } : null;
return d ? { dest, definition: d } : null;
}
}
@@ -83,33 +83,33 @@ function createUiRouter({ rt, auth }) {
@@ -83,33 +83,33 @@ function createUiRouter({ rt, auth }) {
// What shipped on OpenRe.Stream: the shared update log every OpenVibe site has.
// What shipped on OpenRe.Stream: the shared update log every OpenVibe site has.
router.get('/updates', (req, res) => page(req, res, { canonicalPath: '/updates', robots: 'index,follow', title: 'What shipped on OpenRe.Stream', body: frame.updatesBody({ service: 'openre', siteName: 'OpenRe.Stream' }) + `<script src="${ovServe.url('shipped.js')}" defer></script>` }));
router.get('/updates', (req, res) => page(req, res, { canonicalPath: '/updates', robots: 'index,follow', title: 'What shipped on OpenRe.Stream', body: frame.updatesBody({ service: 'openre', siteName: 'OpenRe.Stream' }) + `<script src="${ovServe.url('shipped.js')}" defer></script>` }));
- router.get('/', (req, res) => {
+ router.get('/', async (req, res) => {
const signedIn = req.caller.kind === 'user' && req.caller.subject;
const signedIn = req.caller.kind === 'user' && req.caller.subject;
- const mine = signedIn ? store.definitions.list({ owner_subject: req.caller.subject }) : [];
+ const mine = signedIn ? await store.definitions.list({ owner_subject: req.caller.subject }) : [];
page(req, res, {
page(req, res, {
canonicalPath: '/', robots: 'index,follow',
canonicalPath: '/', robots: 'index,follow',
body: `<h1>OpenRe.Stream</h1>
body: `<h1>OpenRe.Stream</h1>
<p>Ingest and restream for the OpenVibe network: stream definitions with hashed ingest keys, RTMP ingest sessions that run in transport workers separate from any web deploy, restream outputs with health and logs, and recording requests to OpenVibe.Media.</p>
<p>Ingest and restream for the OpenVibe network: stream definitions with hashed ingest keys, RTMP ingest sessions that run in transport workers separate from any web deploy, restream outputs with health and logs, and recording requests to OpenVibe.Media.</p>
<p class="muted">Status: alpha. RTMP ingest and RTMP/SRT restreaming work here; WHIP, WebRTC/SFU and JSMPEG are still served by OpenVibe.Live. Channels, discovery and watch pages stay on <a href="${esc(config.liveUrl)}">openvibe.live</a>.</p>
<p class="muted">Status: alpha. RTMP ingest and RTMP/SRT restreaming work here; WHIP, WebRTC/SFU and JSMPEG are still served by OpenVibe.Live. Channels, discovery and watch pages stay on <a href="${esc(config.liveUrl)}">openvibe.live</a>.</p>
-${signedIn ? `<h2>Your streams</h2>${mine.length ? streamTable(mine) : '<p class="muted">No streams yet.</p>'}<p><a href="/streams">Manage streams</a></p>` : '<p><a href="/auth/login?next=/streams">Sign in with OpenVibe</a> to manage your streams.</p>'}
+${signedIn ? `<h2>Your streams</h2>${mine.length ? await streamTable(mine) : '<p class="muted">No streams yet.</p>'}<p><a href="/streams">Manage streams</a></p>` : '<p><a href="/auth/login?next=/streams">Sign in with OpenVibe</a> to manage your streams.</p>'}
${frame.shipped({ service: 'openre', title: 'Recently shipped on OpenRe.Stream' })}`,
${frame.shipped({ service: 'openre', title: 'Recently shipped on OpenRe.Stream' })}`,
});
});
});
});
- function streamTable(list) {
- return `<table><tr><th>Stream</th><th>State</th><th>Now</th><th>Linked to</th></tr>${list.map((d) => {
- const open = store.sessions.openFor(d.id)[0];
+ async function streamTable(list) {
+ return `<table><tr><th>Stream</th><th>State</th><th>Now</th><th>Linked to</th></tr>${(await Promise.all(list.map(async (d) => {
+ const open = (await store.sessions.openFor(d.id))[0];
const refs = d.external_refs.map(r => `${r.service}:${r.type}:${r.id}`).join(', ');
const refs = d.external_refs.map(r => `${r.service}:${r.type}:${r.id}`).join(', ');
return `<tr><td><a href="/streams/${esc(d.id)}">${esc(d.title)}</a></td><td>${pill(d.state)}</td><td>${open ? pill(open.state) : '<span class="muted">offline</span>'}</td><td class="mono">${esc(refs) || '—'}</td></tr>`;
return `<tr><td><a href="/streams/${esc(d.id)}">${esc(d.title)}</a></td><td>${pill(d.state)}</td><td>${open ? pill(open.state) : '<span class="muted">offline</span>'}</td><td class="mono">${esc(refs) || '—'}</td></tr>`;
- }).join('')}</table>`;
+ }))).join('')}</table>`;
}
}
- router.get('/streams', (req, res) => {
+ router.get('/streams', async (req, res) => {
if (!needUser(req, res)) return;
if (!needUser(req, res)) return;
- const list = store.definitions.list({ owner_subject: req.caller.subject });
+ const list = await store.definitions.list({ owner_subject: req.caller.subject });
page(req, res, {
page(req, res, {
title: 'Streams', canonicalPath: '/streams',
title: 'Streams', canonicalPath: '/streams',
- body: `<h1>Streams</h1>${list.length ? streamTable(list) : '<p class="muted">No streams yet.</p>'}
+ body: `<h1>Streams</h1>${list.length ? await streamTable(list) : '<p class="muted">No streams yet.</p>'}
<div class="card"><h2>New stream</h2>${form(req, '/streams', `
<div class="card"><h2>New stream</h2>${form(req, '/streams', `
<label>Title</label><input type="text" name="title" maxlength="140" required>
<label>Title</label><input type="text" name="title" maxlength="140" required>
<div class="row"><div><label>Recording</label><select name="recording_mode"><option value="vod">Record a VOD</option><option value="clips">Clips only</option><option value="none">Do not record</option></select></div>
<div class="row"><div><label>Recording</label><select name="recording_mode"><option value="vod">Record a VOD</option><option value="clips">Clips only</option><option value="none">Do not record</option></select></div>
@@ -118,8 +118,8 @@ ${frame.shipped({ service: 'openre', title: 'Recently shipped on OpenRe.Stream'
@@ -118,8 +118,8 @@ ${frame.shipped({ service: 'openre', title: 'Recently shipped on OpenRe.Stream'
});
});
});
});
- function keyPage(req, res, definition, key, heading) {
- const ep = store.definitions.ingestEndpoints(definition);
+ async function keyPage(req, res, definition, key, heading) {
+ const ep = await store.definitions.ingestEndpoints(definition);
res.set('Cache-Control', 'no-store');
res.set('Cache-Control', 'no-store');
page(req, res, {
page(req, res, {
title: heading, canonicalPath: `/streams/${definition.id}`,
title: heading, canonicalPath: `/streams/${definition.id}`,
@@ -131,26 +131,26 @@ ${frame.shipped({ service: 'openre', title: 'Recently shipped on OpenRe.Stream'
@@ -131,26 +131,26 @@ ${frame.shipped({ service: 'openre', title: 'Recently shipped on OpenRe.Stream'
});
});
}
}
- post('/streams', (req, res) => {
+ post('/streams', async (req, res) => {
const b = req.body || {};
const b = req.body || {};
- const { definition, key } = store.definitions.create({
+ const { definition, key } = await store.definitions.create({
owner_subject: req.caller.subject, title: b.title, recording_mode: b.recording_mode,
owner_subject: req.caller.subject, title: b.title, recording_mode: b.recording_mode,
recording_visibility: b.recording_visibility, created_by: req.caller.subject,
recording_visibility: b.recording_visibility, created_by: req.caller.subject,
});
});
- keyPage(req, res, definition, key, 'Stream created');
+ await keyPage(req, res, definition, key, 'Stream created');
});
});
- router.get('/streams/:id', (req, res) => {
+ router.get('/streams/:id', async (req, res) => {
if (!needUser(req, res)) return;
if (!needUser(req, res)) return;
- const d = ownDefinition(req, res, req.params.id);
+ const d = await ownDefinition(req, res, req.params.id);
if (!d) return;
if (!d) return;
- const ep = store.definitions.ingestEndpoints(d);
- const keys = store.definitions.keys(d.id).filter(k => k.status !== 'revoked');
- const dests = store.outputs.destinations(d.id);
- const sessions = store.sessions.list({ definition_id: d.id, limit: 15 });
- const open = sessions.find(s => ['starting', 'live', 'ending'].includes(s.state));
- const outputs = open ? store.outputs.outputsOfSession(open.id) : [];
- const outFor = (destId) => outputs.find(o => o.destination_id === destId);
+ const ep = await store.definitions.ingestEndpoints(d);
+ const keys = (await store.definitions.keys(d.id)).filter(k => k.status !== 'revoked');
+ const dests = await store.outputs.destinations(d.id);
+ const sessions = await store.sessions.list({ definition_id: d.id, limit: 15 });
+ const open = await sessions.find(s => ['starting', 'live', 'ending'].includes(s.state));
+ const outputs = open ? await store.outputs.outputsOfSession(open.id) : [];
+ const outFor = async (destId) => await outputs.find(o => o.destination_id === destId);
page(req, res, {
page(req, res, {
title: d.title, canonicalPath: `/streams/${d.id}`,
title: d.title, canonicalPath: `/streams/${d.id}`,
body: `<h1>${esc(d.title)} ${pill(d.state)}</h1>
body: `<h1>${esc(d.title)} ${pill(d.state)}</h1>
@@ -164,14 +164,14 @@ ${form(req, `/streams/${d.id}/rotate`, `<div class="row"><div><label>Old key sta
@@ -164,14 +164,14 @@ ${form(req, `/streams/${d.id}/rotate`, `<div class="row"><div><label>Old key sta
<div class="card"><h2>Now</h2>${open ? `<p>${pill(open.state)} since ${esc(iso(open.live_at || open.created_at))} on ${esc(open.worker_kind)} generation ${esc(open.worker_generation)} · <a href="/sessions/${esc(open.id)}">session</a></p>
<div class="card"><h2>Now</h2>${open ? `<p>${pill(open.state)} since ${esc(iso(open.live_at || open.created_at))} on ${esc(open.worker_kind)} generation ${esc(open.worker_generation)} · <a href="/sessions/${esc(open.id)}">session</a></p>
${form(req, `/sessions/${open.id}/end`, '<button type="submit" class="danger">End session</button>', { cls: 'inline', confirm: 'Disconnect the encoder?' })}` : '<p class="muted">Offline.</p>'}</div>
${form(req, `/sessions/${open.id}/end`, '<button type="submit" class="danger">End session</button>', { cls: 'inline', confirm: 'Disconnect the encoder?' })}` : '<p class="muted">Offline.</p>'}</div>
<div class="card"><h2>Restream destinations</h2>
<div class="card"><h2>Restream destinations</h2>
-${dests.length ? `<table><tr><th>Destination</th><th>Output</th><th>Health</th><th>Actions</th></tr>${dests.map((x) => {
- const o = outFor(x.id);
+${dests.length ? `<table><tr><th>Destination</th><th>Output</th><th>Health</th><th>Actions</th></tr>${(await Promise.all(dests.map(async (x) => {
+ const o = await outFor(x.id);
return `<tr><td><strong>${esc(x.name || x.platform)}</strong> <span class="muted">${esc(x.platform)}</span><br><span class="mono">${esc(x.server_url)}</span><br><span class="muted">key ${esc(x.stream_key_hint || 'not set')}${x.enabled ? '' : ' · disabled'}${x.auto_start ? ' · auto-start' : ''}${x.hold_reason ? ` · held: ${esc(x.hold_reason)}` : ''}${x.cooldown_ms ? ` · cooling down until ${esc(x.cooldown_until)}` : ''}</span></td>
return `<tr><td><strong>${esc(x.name || x.platform)}</strong> <span class="muted">${esc(x.platform)}</span><br><span class="mono">${esc(x.server_url)}</span><br><span class="muted">key ${esc(x.stream_key_hint || 'not set')}${x.enabled ? '' : ' · disabled'}${x.auto_start ? ' · auto-start' : ''}${x.hold_reason ? ` · held: ${esc(x.hold_reason)}` : ''}${x.cooldown_ms ? ` · cooling down until ${esc(x.cooldown_until)}` : ''}</span></td>
<td>${o ? pill(o.state) : '<span class="muted">—</span>'}</td>
<td>${o ? pill(o.state) : '<span class="muted">—</span>'}</td>
<td>${o && o.progress ? `${esc(o.progress.fps ?? '?')} fps · ${esc(o.progress.bitrate_kbps ?? '?')} kbit/s · speed ${esc(o.progress.speed ?? '?')}` : ''}${o && o.last_error ? `<br><span class="muted">${esc(o.last_error)}</span>` : ''}${!o && x.last_error ? `<span class="muted">${esc(x.last_error)}</span>` : ''}</td>
<td>${o && o.progress ? `${esc(o.progress.fps ?? '?')} fps · ${esc(o.progress.bitrate_kbps ?? '?')} kbit/s · speed ${esc(o.progress.speed ?? '?')}` : ''}${o && o.last_error ? `<br><span class="muted">${esc(o.last_error)}</span>` : ''}${!o && x.last_error ? `<span class="muted">${esc(x.last_error)}</span>` : ''}</td>
<td>${form(req, `/destinations/${x.id}/test`, '<button type="submit" class="secondary">Test</button>', { cls: 'inline' })}${open ? (o && ['pending', 'starting', 'live', 'error'].includes(o.state) ? form(req, `/destinations/${x.id}/stop`, '<button type="submit" class="secondary">Stop</button>', { cls: 'inline' }) : form(req, `/destinations/${x.id}/start`, '<button type="submit">Start</button>', { cls: 'inline' })) : ''}
<td>${form(req, `/destinations/${x.id}/test`, '<button type="submit" class="secondary">Test</button>', { cls: 'inline' })}${open ? (o && ['pending', 'starting', 'live', 'error'].includes(o.state) ? form(req, `/destinations/${x.id}/stop`, '<button type="submit" class="secondary">Stop</button>', { cls: 'inline' }) : form(req, `/destinations/${x.id}/start`, '<button type="submit">Start</button>', { cls: 'inline' })) : ''}
<a href="/destinations/${esc(x.id)}">Edit</a>${form(req, `/destinations/${x.id}/delete`, '<button type="submit" class="danger">Delete</button>', { cls: 'inline', confirm: 'Delete this destination?' })}</td></tr>`;
<a href="/destinations/${esc(x.id)}">Edit</a>${form(req, `/destinations/${x.id}/delete`, '<button type="submit" class="danger">Delete</button>', { cls: 'inline', confirm: 'Delete this destination?' })}</td></tr>`;
- }).join('')}</table>` : '<p class="muted">No destinations.</p>'}
+ }))).join('')}</table>` : '<p class="muted">No destinations.</p>'}
<h3>Add a destination</h3>${form(req, `/streams/${d.id}/destinations`, destinationFields({}))}</div>
<h3>Add a destination</h3>${form(req, `/streams/${d.id}/destinations`, destinationFields({}))}</div>
<div class="card"><h2>Settings</h2>${form(req, `/streams/${d.id}`, `
<div class="card"><h2>Settings</h2>${form(req, `/streams/${d.id}`, `
<label>Title</label><input type="text" name="title" maxlength="140" value="${esc(d.title)}">
<label>Title</label><input type="text" name="title" maxlength="140" value="${esc(d.title)}">
@@ -215,35 +215,35 @@ ${dests.length ? `<table><tr><th>Destination</th><th>Output</th><th>Health</th><
@@ -215,35 +215,35 @@ ${dests.length ? `<table><tr><th>Destination</th><th>Output</th><th>Health</th><
return `<table><tr><th>Session</th><th>State</th><th>Started</th><th>Ended</th><th>Worker</th></tr>${list.map(s => `<tr><td><a class="mono" href="/sessions/${esc(s.id)}">${esc(s.id)}</a></td><td>${pill(s.state)}${s.failure_reason ? ` <span class="muted">${esc(s.failure_reason)}</span>` : ''}</td><td>${esc(iso(s.live_at || s.created_at))}</td><td>${esc(iso(s.ended_at))}</td><td>${esc(s.worker_kind)} #${esc(s.worker_generation)}</td></tr>`).join('')}</table>`;
return `<table><tr><th>Session</th><th>State</th><th>Started</th><th>Ended</th><th>Worker</th></tr>${list.map(s => `<tr><td><a class="mono" href="/sessions/${esc(s.id)}">${esc(s.id)}</a></td><td>${pill(s.state)}${s.failure_reason ? ` <span class="muted">${esc(s.failure_reason)}</span>` : ''}</td><td>${esc(iso(s.live_at || s.created_at))}</td><td>${esc(iso(s.ended_at))}</td><td>${esc(s.worker_kind)} #${esc(s.worker_generation)}</td></tr>`).join('')}</table>`;
}
}
- post('/streams/:id', (req, res) => {
- const d = ownDefinition(req, res, req.params.id);
+ post('/streams/:id', async (req, res) => {
+ const d = await ownDefinition(req, res, req.params.id);
if (!d) return;
if (!d) return;
const b = req.body || {};
const b = req.body || {};
- store.definitions.update(d.id, { title: b.title, recording_mode: b.recording_mode, recording_visibility: b.recording_visibility, playback_visibility: b.playback_visibility, state: b.state, mirror_to_live: b.mirror_to_live === '1' });
+ await store.definitions.update(d.id, { title: b.title, recording_mode: b.recording_mode, recording_visibility: b.recording_visibility, playback_visibility: b.playback_visibility, state: b.state, mirror_to_live: b.mirror_to_live === '1' });
res.redirect(303, `/streams/${d.id}`);
res.redirect(303, `/streams/${d.id}`);
});
});
- post('/streams/:id/rotate', (req, res) => {
- const d = ownDefinition(req, res, req.params.id);
+ post('/streams/:id/rotate', async (req, res) => {
+ const d = await ownDefinition(req, res, req.params.id);
if (!d) return;
if (!d) return;
- const r = store.definitions.rotateKey(d.id, { grace_seconds: Number(req.body.grace_seconds) || 0, rotated_by: req.caller.subject });
- if (req.body.end_sessions === '1') for (const s of store.sessions.openFor(d.id)) store.sessions.requestEnd(s.id, `key_rotation:${req.caller.subject}`);
- keyPage(req, res, store.definitions.get(d.id), r.key, 'New ingest key');
+ const r = await store.definitions.rotateKey(d.id, { grace_seconds: Number(req.body.grace_seconds) || 0, rotated_by: req.caller.subject });
+ if (req.body.end_sessions === '1') for (const s of await store.sessions.openFor(d.id)) await store.sessions.requestEnd(s.id, `key_rotation:${req.caller.subject}`);
+ await keyPage(req, res, await store.definitions.get(d.id), r.key, 'New ingest key');
});
});
- post('/streams/:id/destinations', (req, res) => {
- const d = ownDefinition(req, res, req.params.id);
+ post('/streams/:id/destinations', async (req, res) => {
+ const d = await ownDefinition(req, res, req.params.id);
if (!d) return;
if (!d) return;
- store.outputs.createDestination(d.id, destInput(req.body || {}, { editing: false }));
+ await store.outputs.createDestination(d.id, destInput(req.body || {}, { editing: false }));
res.redirect(303, `/streams/${d.id}`);
res.redirect(303, `/streams/${d.id}`);
});
});
- router.get('/destinations/:id', (req, res) => {
+ router.get('/destinations/:id', async (req, res) => {
if (!needUser(req, res)) return;
if (!needUser(req, res)) return;
- const own = ownDestination(req, res, req.params.id);
+ const own = await ownDestination(req, res, req.params.id);
if (!own) return;
if (!own) return;
const x = store.outputs.publicDest(own.dest);
const x = store.outputs.publicDest(own.dest);
- const logs = store.outputs.logsOfDestination(x.id, 50);
+ const logs = await store.outputs.logsOfDestination(x.id, 50);
page(req, res, {
page(req, res, {
title: `Destination ${x.name || x.platform}`, canonicalPath: `/destinations/${x.id}`,
title: `Destination ${x.name || x.platform}`, canonicalPath: `/destinations/${x.id}`,
body: `<h1>${esc(x.name || x.platform)}</h1><p><a href="/streams/${esc(own.definition.id)}">← ${esc(own.definition.title)}</a></p>
body: `<h1>${esc(x.name || x.platform)}</h1><p><a href="/streams/${esc(own.definition.id)}">← ${esc(own.definition.title)}</a></p>
@@ -252,25 +252,25 @@ ${dests.length ? `<table><tr><th>Destination</th><th>Output</th><th>Health</th><
@@ -252,25 +252,25 @@ ${dests.length ? `<table><tr><th>Destination</th><th>Output</th><th>Health</th><
});
});
});
});
- post('/destinations/:id', (req, res) => {
- const own = ownDestination(req, res, req.params.id);
+ post('/destinations/:id', async (req, res) => {
+ const own = await ownDestination(req, res, req.params.id);
if (!own) return;
if (!own) return;
- store.outputs.updateDestination(own.dest.id, destInput(req.body || {}, { editing: true }));
+ await store.outputs.updateDestination(own.dest.id, destInput(req.body || {}, { editing: true }));
res.redirect(303, `/streams/${own.definition.id}`);
res.redirect(303, `/streams/${own.definition.id}`);
});
});
- post('/destinations/:id/delete', (req, res) => {
- const own = ownDestination(req, res, req.params.id);
+ post('/destinations/:id/delete', async (req, res) => {
+ const own = await ownDestination(req, res, req.params.id);
if (!own) return;
if (!own) return;
- store.outputs.deleteDestination(own.dest.id);
+ await store.outputs.deleteDestination(own.dest.id);
res.redirect(303, `/streams/${own.definition.id}`);
res.redirect(303, `/streams/${own.definition.id}`);
});
});
post('/destinations/:id/test', async (req, res) => {
post('/destinations/:id/test', async (req, res) => {
- const own = ownDestination(req, res, req.params.id);
+ const own = await ownDestination(req, res, req.params.id);
if (!own) return;
if (!own) return;
const result = await testDestination(own.dest, { allowPrivate: config.outputs.allowPrivateHosts });
const result = await testDestination(own.dest, { allowPrivate: config.outputs.allowPrivateHosts });
- store.outputs.log(null, own.dest.id, result.ok ? 'info' : 'warn', `test ${result.ok ? 'passed' : 'failed'}: ${result.checks.map(c => `${c.check} ${c.ok ? 'ok' : 'FAIL'} (${c.detail})`).join('; ')}`);
+ await store.outputs.log(null, own.dest.id, result.ok ? 'info' : 'warn', `test ${result.ok ? 'passed' : 'failed'}: ${result.checks.map(c => `${c.check} ${c.ok ? 'ok' : 'FAIL'} (${c.detail})`).join('; ')}`);
page(req, res, {
page(req, res, {
title: 'Destination test',
title: 'Destination test',
body: `<h1>Test ${result.ok ? pill('ok') : pill('failed')}</h1><table>${result.checks.map(c => `<tr><td>${esc(c.check)}</td><td>${c.ok ? pill('ok') : pill('failed')}</td><td>${esc(c.detail)}</td></tr>`).join('')}</table>
body: `<h1>Test ${result.ok ? pill('ok') : pill('failed')}</h1><table>${result.checks.map(c => `<tr><td>${esc(c.check)}</td><td>${c.ok ? pill('ok') : pill('failed')}</td><td>${esc(c.detail)}</td></tr>`).join('')}</table>
@@ -278,34 +278,34 @@ ${dests.length ? `<table><tr><th>Destination</th><th>Output</th><th>Health</th><
@@ -278,34 +278,34 @@ ${dests.length ? `<table><tr><th>Destination</th><th>Output</th><th>Health</th><
});
});
});
});
- post('/destinations/:id/start', (req, res) => {
- const own = ownDestination(req, res, req.params.id);
+ post('/destinations/:id/start', async (req, res) => {
+ const own = await ownDestination(req, res, req.params.id);
if (!own) return;
if (!own) return;
- store.outputs.startDestination(own.dest.id);
+ await store.outputs.startDestination(own.dest.id);
res.redirect(303, `/streams/${own.definition.id}`);
res.redirect(303, `/streams/${own.definition.id}`);
});
});
- post('/destinations/:id/stop', (req, res) => {
- const own = ownDestination(req, res, req.params.id);
+ post('/destinations/:id/stop', async (req, res) => {
+ const own = await ownDestination(req, res, req.params.id);
if (!own) return;
if (!own) return;
- store.outputs.stopDestination(own.dest.id);
+ await store.outputs.stopDestination(own.dest.id);
res.redirect(303, `/streams/${own.definition.id}`);
res.redirect(303, `/streams/${own.definition.id}`);
});
});
- router.get('/sessions', (req, res) => {
+ router.get('/sessions', async (req, res) => {
if (!needUser(req, res)) return;
if (!needUser(req, res)) return;
const all = req.caller.staff && req.query.all === '1';
const all = req.caller.staff && req.query.all === '1';
- page(req, res, { title: 'Sessions', canonicalPath: '/sessions', body: `<h1>Sessions</h1>${req.caller.staff ? `<p class="muted"><a href="/sessions${all ? '' : '?all=1'}">${all ? 'Only mine' : 'Everyone (staff)'}</a></p>` : ''}${sessionTable(store.sessions.list({ owner_subject: all ? undefined : req.caller.subject, limit: 100 }))}` });
+ page(req, res, { title: 'Sessions', canonicalPath: '/sessions', body: `<h1>Sessions</h1>${req.caller.staff ? `<p class="muted"><a href="/sessions${all ? '' : '?all=1'}">${all ? 'Only mine' : 'Everyone (staff)'}</a></p>` : ''}${sessionTable(await store.sessions.list({ owner_subject: all ? undefined : req.caller.subject, limit: 100 }))}` });
});
});
- router.get('/sessions/:id', (req, res) => {
+ router.get('/sessions/:id', async (req, res) => {
if (!needUser(req, res)) return;
if (!needUser(req, res)) return;
- const s = store.sessions.get(req.params.id);
- const d = s ? store.definitions.row(s.definition_id) : null;
+ const s = await store.sessions.get(req.params.id);
+ const d = s ? await store.definitions.row(s.definition_id) : null;
if (!s || !d || !auth.canAccess(req.caller, d.owner_subject)) return page(req, res, { title: 'Not found', body: '<h1>Session not found</h1>' }, 404);
if (!s || !d || !auth.canAccess(req.caller, d.owner_subject)) return page(req, res, { title: 'Not found', body: '<h1>Session not found</h1>' }, 404);
- const outputs = store.outputs.outputsOfSession(s.id);
- const rec = store.recordings.bySession(s.id);
- const pb = store.sessions.playback(s);
+ const outputs = await store.outputs.outputsOfSession(s.id);
+ const rec = await store.recordings.bySession(s.id);
+ const pb = await store.sessions.playback(s);
const open = ['starting', 'live', 'ending'].includes(s.state);
const open = ['starting', 'live', 'ending'].includes(s.state);
return page(req, res, {
return page(req, res, {
title: `Session ${s.id}`, canonicalPath: `/sessions/${s.id}`,
title: `Session ${s.id}`, canonicalPath: `/sessions/${s.id}`,
@@ -318,16 +318,16 @@ ${dests.length ? `<table><tr><th>Destination</th><th>Output</th><th>Health</th><
@@ -318,16 +318,16 @@ ${dests.length ? `<table><tr><th>Destination</th><th>Output</th><th>Health</th><
<tr><th>Recording</th><td>${rec ? `${pill(rec.state)} ${rec.media_vod_id ? `Media ${esc(rec.media_app)} VOD ${esc(rec.media_vod_id)}` : ''} ${esc(rec.last_error || '')}` : '<span class="muted">none</span>'}</td></tr>
<tr><th>Recording</th><td>${rec ? `${pill(rec.state)} ${rec.media_vod_id ? `Media ${esc(rec.media_app)} VOD ${esc(rec.media_vod_id)}` : ''} ${esc(rec.last_error || '')}` : '<span class="muted">none</span>'}</td></tr>
<tr><th>Playback</th><td>${pb && pb.flv && s.state === 'live' ? `<a class="mono" href="${esc(pb.flv.public_url)}">${esc(pb.flv.public_url)}</a> (HTTP-FLV)` : '—'}</td></tr>
<tr><th>Playback</th><td>${pb && pb.flv && s.state === 'live' ? `<a class="mono" href="${esc(pb.flv.public_url)}">${esc(pb.flv.public_url)}</a> (HTTP-FLV)` : '—'}</td></tr>
</table>${open ? form(req, `/sessions/${s.id}/end`, '<button type="submit" class="danger">End session</button>', { confirm: 'Disconnect the encoder?' }) : ''}</div>
</table>${open ? form(req, `/sessions/${s.id}/end`, '<button type="submit" class="danger">End session</button>', { confirm: 'Disconnect the encoder?' }) : ''}</div>
-<h2>Outputs</h2>${outputs.length ? `<table><tr><th>Output</th><th>State</th><th>Health</th><th>Restarts</th></tr>${outputs.map(o => `<tr><td class="mono">${esc(o.id)}<br><span class="muted">${esc(o.destination_id)}</span></td><td>${pill(o.state)}</td><td>${o.progress ? `${esc(o.progress.fps ?? '?')} fps · ${esc(o.progress.bitrate_kbps ?? '?')} kbit/s` : ''} ${esc(o.last_error || '')}</td><td>${esc(o.restart_attempts)}/${esc(o.max_restart_attempts)}</td></tr>`).join('')}</table>` : '<p class="muted">No outputs.</p>'}
-<h2>Lifecycle</h2><table>${store.sessions.transitions(s.id).map(t => `<tr><td>${esc(iso(t.at))}</td><td>${esc(t.from_state || '·')} → ${pill(t.to_state)}</td><td>${esc(t.reason || '')}</td><td class="mono muted">${esc(t.actor || '')}</td></tr>`).join('')}</table>`,
+<h2>Outputs</h2>${outputs.length ? `<table><tr><th>Output</th><th>State</th><th>Health</th><th>Restarts</th></tr>${(await outputs.map(o => `<tr><td class="mono">${esc(o.id)}<br><span class="muted">${esc(o.destination_id)}</span></td><td>${pill(o.state)}</td><td>${o.progress ? `${esc(o.progress.fps ?? '?')} fps · ${esc(o.progress.bitrate_kbps ?? '?')} kbit/s` : ''} ${esc(o.last_error || '')}</td><td>${esc(o.restart_attempts)}/${esc(o.max_restart_attempts)}</td></tr>`)).join('')}</table>` : '<p class="muted">No outputs.</p>'}
+<h2>Lifecycle</h2><table>${(await store.sessions.transitions(s.id)).map(t => `<tr><td>${esc(iso(t.at))}</td><td>${esc(t.from_state || '·')} → ${pill(t.to_state)}</td><td>${esc(t.reason || '')}</td><td class="mono muted">${esc(t.actor || '')}</td></tr>`).join('')}</table>`,
});
});
});
});
- post('/sessions/:id/end', (req, res) => {
- const s = store.sessions.get(req.params.id);
- const d = s ? store.definitions.row(s.definition_id) : null;
+ post('/sessions/:id/end', async (req, res) => {
+ const s = await store.sessions.get(req.params.id);
+ const d = s ? await store.definitions.row(s.definition_id) : null;
if (!s || !d || !auth.canAccess(req.caller, d.owner_subject)) return page(req, res, { title: 'Not found', body: '<h1>Session not found</h1>' }, 404);
if (!s || !d || !auth.canAccess(req.caller, d.owner_subject)) return page(req, res, { title: 'Not found', body: '<h1>Session not found</h1>' }, 404);
- store.sessions.requestEnd(s.id, req.caller.subject);
+ await store.sessions.requestEnd(s.id, req.caller.subject);
return res.redirect(303, `/sessions/${s.id}`);
return res.redirect(303, `/sessions/${s.id}`);
});
});
@@ -5,10 +5,14 @@
@@ -5,10 +5,14 @@
// by the worker; a new API process starts and serves the same live session.
// by the worker; a new API process starts and serves the same live session.
const assert = require('assert');
const assert = require('assert');
const path = require('path');
const path = require('path');
-const { tmpDir, testEnv, child, waitFor, request, userToken, freePort, sleep, suite, OWNER } = require('./helpers');
+const { tmpDir, testEnv, child, waitFor, request, userToken, freePort, sleep, suite, OWNER, multiProcess } = require('./helpers');
const { load } = require('../server/config');
const { load } = require('../server/config');
const { openRuntime } = require('../server/store');
const { openRuntime } = require('../server/store');
+if (!multiProcess()) {
+ console.log('api-restart: skipped (OpenRe processes sharing one database need PostgreSQL: npm run test:pg)');
+ process.exit(0);
+}
const t = suite('api-restart');
const t = suite('api-restart');
const dir = tmpDir();
const dir = tmpDir();
let env;
let env;
@@ -26,9 +30,9 @@ async function startApi() {
@@ -26,9 +30,9 @@ async function startApi() {
}
}
t('setup: API process + fake worker process holding a live session', async () => {
t('setup: API process + fake worker process holding a live session', async () => {
- env = testEnv(dir, { PORT: String(await freePort()), OPENRE_WORKER_HEARTBEAT_MS: '200', OPENRE_WORKER_LEASE_MS: '1500' });
- const rt = openRuntime({ config: load(env), log: { log() {}, warn() {}, error() {} } });
- const { definition } = rt.store.definitions.create({ owner_subject: OWNER, title: 'Restart test' });
+ env = await testEnv(dir, { PORT: String(await freePort()), OPENRE_WORKER_HEARTBEAT_MS: '200', OPENRE_WORKER_LEASE_MS: '1500' });
+ const rt = await openRuntime({ config: load(env), log: { log() {}, warn() {}, error() {} } });
+ const { definition } = await rt.store.definitions.create({ owner_subject: OWNER, title: 'Restart test' });
rt.db.close();
rt.db.close();
api = await startApi();
api = await startApi();
fake = child(path.join('test', 'fixtures', 'fake-worker.js'), { ...env, FAKE_STREAM_ID: definition.id });
fake = child(path.join('test', 'fixtures', 'fake-worker.js'), { ...env, FAKE_STREAM_ID: definition.id });
@@ -39,18 +43,18 @@ t('setup: API process + fake worker process holding a live session', async () =>
@@ -39,18 +43,18 @@ t('setup: API process + fake worker process holding a live session', async () =>
});
});
t('stopping the API (SIGTERM) leaves the worker process and its session alone', async () => {
t('stopping the API (SIGTERM) leaves the worker process and its session alone', async () => {
- const rt = openRuntime({ config: load(env), log: { log() {}, warn() {}, error() {} } });
- const leaseBefore = rt.store.sessions.get(sessionId).lease_expires_at;
+ const rt = await openRuntime({ config: load(env), log: { log() {}, warn() {}, error() {} } });
+ const leaseBefore = (await rt.store.sessions.get(sessionId)).lease_expires_at;
api.proc.kill('SIGTERM');
api.proc.kill('SIGTERM');
const exit = await api.proc.exited;
const exit = await api.proc.exited;
assert.strictEqual(exit.code, 0, 'the API exits cleanly');
assert.strictEqual(exit.code, 0, 'the API exits cleanly');
// Longer than a lease: if anything depended on the API, the session would now be failing.
// Longer than a lease: if anything depended on the API, the session would now be failing.
await sleep(2500);
await sleep(2500);
assert.strictEqual(fake.exitCode, null, 'fake worker still running');
assert.strictEqual(fake.exitCode, null, 'fake worker still running');
- const s = rt.store.sessions.get(sessionId);
+ const s = await rt.store.sessions.get(sessionId);
assert.strictEqual(s.state, 'live');
assert.strictEqual(s.state, 'live');
assert.ok(s.lease_expires_at > leaseBefore, 'the worker kept renewing its lease while the API was down');
assert.ok(s.lease_expires_at > leaseBefore, 'the worker kept renewing its lease while the API was down');
- assert.strictEqual(rt.store.workers.get(workerId).state, 'ready');
+ assert.strictEqual((await rt.store.workers.get(workerId)).state, 'ready');
rt.db.close();
rt.db.close();
});
});
@@ -88,8 +88,8 @@ t('rotation over HTTP returns the new key once and the old one stops authenticat
@@ -88,8 +88,8 @@ t('rotation over HTTP returns the new key once and the old one stops authenticat
assert.strictEqual(r.status, 200);
assert.strictEqual(r.status, 200);
assert.notStrictEqual(r.body.key.key, firstKey);
assert.notStrictEqual(r.body.key.key, firstKey);
assert.strictEqual(r.body.retired[0].status, 'revoked');
assert.strictEqual(r.body.retired[0].status, 'revoked');
- assert.strictEqual(api.rt.store.definitions.resolveIngestKey(firstKey, 'rtmp').error, 'revoked_key');
- assert.ok(api.rt.store.definitions.resolveIngestKey(r.body.key.key, 'rtmp').definition);
+ assert.strictEqual((await api.rt.store.definitions.resolveIngestKey(firstKey, 'rtmp')).error, 'revoked_key');
+ assert.ok((await api.rt.store.definitions.resolveIngestKey(r.body.key.key, 'rtmp')).definition);
});
});
t('destinations: validated, masked, never returned in full; SSRF rules apply', async () => {
t('destinations: validated, masked, never returned in full; SSRF rules apply', async () => {
@@ -129,13 +129,13 @@ t('the destination test reports URL, DNS and TCP checks without pushing media',
@@ -129,13 +129,13 @@ t('the destination test reports URL, DNS and TCP checks without pushing media',
t('sessions: list, detail with playback descriptor, end request', async () => {
t('sessions: list, detail with playback descriptor, end request', async () => {
const rt = api.rt;
const rt = api.rt;
- const w = rt.store.workers.register({ kind: 'rtmp-ingest', endpoints: { publicPort: 1936, rtmpPlayPort: 19370, flvPort: 19371 } });
- rt.store.workers.ready(w.id);
- const def = rt.store.definitions.get(streamId);
- const a = rt.store.sessions.admit({ definition: def, key: null, protocol: 'rtmp', worker: rt.store.workers.get(w.id) });
- rt.store.sessions.transition(a.session.id, 'live');
+ const w = await rt.store.workers.register({ kind: 'rtmp-ingest', endpoints: { publicPort: 1936, rtmpPlayPort: 19370, flvPort: 19371 } });
+ await rt.store.workers.ready(w.id);
+ const def = await rt.store.definitions.get(streamId);
+ const a = await rt.store.sessions.admit({ definition: def, key: null, protocol: 'rtmp', worker: await rt.store.workers.get(w.id) });
+ await rt.store.sessions.transition(a.session.id, 'live');
const list = await request(api.base, 'GET', '/api/v1/sessions?state=live', { token: owner });
const list = await request(api.base, 'GET', '/api/v1/sessions?state=live', { token: owner });
- assert.deepStrictEqual(list.body.sessions.map(s => s.id), [a.session.id]);
+ assert.deepStrictEqual(await list.body.sessions.map(s => s.id), [a.session.id]);
const detail = await request(api.base, 'GET', `/api/v1/sessions/${a.session.id}`, { token: owner });
const detail = await request(api.base, 'GET', `/api/v1/sessions/${a.session.id}`, { token: owner });
assert.strictEqual(detail.body.session.playback.flv.internal_url, `http://127.0.0.1:19371/live/${a.session.id}.flv`);
assert.strictEqual(detail.body.session.playback.flv.internal_url, `http://127.0.0.1:19371/live/${a.session.id}.flv`);
assert.strictEqual(detail.body.session.playback.flv.public_url, `${api.config.baseUrl}/play/${a.session.id}.flv`);
assert.strictEqual(detail.body.session.playback.flv.public_url, `${api.config.baseUrl}/play/${a.session.id}.flv`);
@@ -145,11 +145,11 @@ t('sessions: list, detail with playback descriptor, end request', async () => {
@@ -145,11 +145,11 @@ t('sessions: list, detail with playback descriptor, end request', async () => {
assert.strictEqual(noEndCap.status, 403);
assert.strictEqual(noEndCap.status, 403);
const end = await request(api.base, 'POST', `/api/v1/sessions/${a.session.id}/end`, { token: owner });
const end = await request(api.base, 'POST', `/api/v1/sessions/${a.session.id}/end`, { token: owner });
assert.strictEqual(end.status, 202);
assert.strictEqual(end.status, 202);
- assert.strictEqual(rt.store.sessions.get(a.session.id).desired_state, 'end');
+ assert.strictEqual((await rt.store.sessions.get(a.session.id)).desired_state, 'end');
// Archiving is refused while it is open.
// Archiving is refused while it is open.
const del = await request(api.base, 'DELETE', `/api/v1/streams/${streamId}`, { token: owner });
const del = await request(api.base, 'DELETE', `/api/v1/streams/${streamId}`, { token: owner });
assert.strictEqual(del.status, 409);
assert.strictEqual(del.status, 409);
- rt.store.sessions.finish(a.session.id, { reason: 'test' });
+ await rt.store.sessions.finish(a.session.id, { reason: 'test' });
assert.strictEqual((await request(api.base, 'GET', `/play/${a.session.id}.flv`)).status, 404, 'no playback for an ended session');
assert.strictEqual((await request(api.base, 'GET', `/play/${a.session.id}.flv`)).status, 404, 'no playback for an ended session');
});
});
@@ -9,8 +9,12 @@ const { load } = require('../server/config');
@@ -9,8 +9,12 @@ const { load } = require('../server/config');
const { openRuntime } = require('../server/store');
const { openRuntime } = require('../server/store');
const { createCoordinator } = require('../server/coordinator');
const { createCoordinator } = require('../server/coordinator');
const { localAddresses, rtmpProbe } = require('../scripts/cutover-preflight');
const { localAddresses, rtmpProbe } = require('../scripts/cutover-preflight');
-const { tmpDir, testEnv, child, waitFor, freePort, suite, silent } = require('./helpers');
+const { tmpDir, testEnv, child, waitFor, freePort, suite, silent, multiProcess } = require('./helpers');
+if (!multiProcess()) {
+ console.log('bind-switch: skipped (OpenRe processes sharing one database need PostgreSQL: npm run test:pg)');
+ process.exit(0);
+}
const t = suite('bind-switch');
const t = suite('bind-switch');
const procs = [];
const procs = [];
process.on('exit', () => { for (const p of procs) { try { p.kill('SIGKILL'); } catch { /* gone */ } } });
process.on('exit', () => { for (const p of procs) { try { p.kill('SIGKILL'); } catch { /* gone */ } } });
@@ -33,7 +37,7 @@ function startWorker(bind) {
@@ -33,7 +37,7 @@ function startWorker(bind) {
t('generation 1 on 127.0.0.1 only: the host\'s other addresses are refused', async () => {
t('generation 1 on 127.0.0.1 only: the host\'s other addresses are refused', async () => {
port = await freePort();
port = await freePort();
const internal = await freePort();
const internal = await freePort();
- env = testEnv(tmpDir(), {
+ env = await testEnv(tmpDir(), {
OPENRE_RTMP_PORT: String(port),
OPENRE_RTMP_PORT: String(port),
OPENRE_RTMP_INTERNAL_PORT_MIN: String(internal),
OPENRE_RTMP_INTERNAL_PORT_MIN: String(internal),
OPENRE_RTMP_INTERNAL_PORT_MAX: String(internal + 20),
OPENRE_RTMP_INTERNAL_PORT_MAX: String(internal + 20),
@@ -57,14 +61,14 @@ t('generation 2 starts with OPENRE_RTMP_BIND=0.0.0.0 on the same port while gene
@@ -57,14 +61,14 @@ t('generation 2 starts with OPENRE_RTMP_BIND=0.0.0.0 on the same port while gene
});
});
t('the coordinator drains generation 1; it closes its listener and exits 0 by itself', async () => {
t('the coordinator drains generation 1; it closes its listener and exits 0 by itself', async () => {
- rt = openRuntime({ config: load(env), log: silent });
+ rt = await openRuntime({ config: load(env), log: silent });
const coordinator = createCoordinator({ rt, media: null, log: silent });
const coordinator = createCoordinator({ rt, media: null, log: silent });
- const out = coordinator.syncTick();
+ const out = await coordinator.syncTick();
assert.strictEqual(out.drained, 1);
assert.strictEqual(out.drained, 1);
const r = await g1.exited;
const r = await g1.exited;
assert.strictEqual(r.code, 0, 'a drained, idle generation exits cleanly');
assert.strictEqual(r.code, 0, 'a drained, idle generation exits cleanly');
assert.match(g1.output, /public listener closed/);
assert.match(g1.output, /public listener closed/);
- const rows = rt.db.prepare("SELECT generation, state, stop_reason FROM workers WHERE kind = 'rtmp-ingest' ORDER BY generation").all();
+ const rows = await rt.db.prepare("SELECT generation, state, stop_reason FROM workers WHERE kind = 'rtmp-ingest' ORDER BY generation").all();
assert.deepStrictEqual(rows.map((w) => [w.generation, w.state]), [[1, 'stopped'], [2, 'ready']]);
assert.deepStrictEqual(rows.map((w) => [w.generation, w.state]), [[1, 'stopped'], [2, 'ready']]);
});
});
@@ -8,130 +8,130 @@ const { createWorkerRuntime } = require('../workers/runtime');
@@ -8,130 +8,130 @@ const { createWorkerRuntime } = require('../workers/runtime');
const t = suite('coordinator');
const t = suite('coordinator');
-function setup() {
+async function setup() {
const clock = manualClock();
const clock = manualClock();
- const rt = runtime({ clock });
+ const rt = await runtime({ clock });
const coordinator = createCoordinator({ rt, media: null, log: silent });
const coordinator = createCoordinator({ rt, media: null, log: silent });
- const { definition, key } = rt.store.definitions.create({ owner_subject: OWNER });
- const worker = (kind) => { const w = rt.store.workers.register({ kind, endpoints: { flvPort: 19361, rtmpPlayPort: 19360 } }); rt.store.workers.ready(w.id); return rt.store.workers.get(w.id); };
+ const { definition, key } = await rt.store.definitions.create({ owner_subject: OWNER });
+ const worker = async (kind) => { const w = await rt.store.workers.register({ kind, endpoints: { flvPort: 19361, rtmpPlayPort: 19360 } }); await rt.store.workers.ready(w.id); return await rt.store.workers.get(w.id); };
return { rt, clock, coordinator, definition, key, worker };
return { rt, clock, coordinator, definition, key, worker };
}
}
-t('generations grow per kind; when a newer one is ready every older one drains with a deadline', () => {
- const { rt, clock, coordinator, worker } = setup();
- const g1 = worker('rtmp-ingest');
- const r1 = worker('restream');
+t('generations grow per kind; when a newer one is ready every older one drains with a deadline', async () => {
+ const { rt, clock, coordinator, worker } = await setup();
+ const g1 = await worker('rtmp-ingest');
+ const r1 = await worker('restream');
assert.strictEqual(g1.generation, 1);
assert.strictEqual(g1.generation, 1);
assert.strictEqual(r1.generation, 1);
assert.strictEqual(r1.generation, 1);
- assert.strictEqual(coordinator.syncTick().drained, 0);
- const g2 = worker('rtmp-ingest');
+ assert.strictEqual((await coordinator.syncTick()).drained, 0);
+ const g2 = await worker('rtmp-ingest');
assert.strictEqual(g2.generation, 2);
assert.strictEqual(g2.generation, 2);
- assert.strictEqual(coordinator.syncTick().drained, 1);
- const old = rt.store.workers.get(g1.id);
+ assert.strictEqual((await coordinator.syncTick()).drained, 1);
+ const old = await rt.store.workers.get(g1.id);
assert.strictEqual(old.state, 'draining');
assert.strictEqual(old.state, 'draining');
assert.strictEqual(old.drain_deadline, clock.now() + rt.config.workers.drainMaxMs);
assert.strictEqual(old.drain_deadline, clock.now() + rt.config.workers.drainMaxMs);
- assert.strictEqual(rt.store.workers.get(r1.id).state, 'ready', 'other kinds are independent');
- assert.strictEqual(rt.store.workers.newestReady('rtmp-ingest').id, g2.id);
+ assert.strictEqual((await rt.store.workers.get(r1.id)).state, 'ready', 'other kinds are independent');
+ assert.strictEqual((await rt.store.workers.newestReady('rtmp-ingest')).id, g2.id);
});
});
-t('a draining generation keeps its live session; new sessions only reach the newest generation', () => {
- const { rt, coordinator, definition, key, worker } = setup();
- const g1 = worker('rtmp-ingest');
- const a = rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker: g1 });
- rt.store.sessions.transition(a.session.id, 'live');
- worker('rtmp-ingest');
- coordinator.syncTick();
- assert.strictEqual(rt.store.sessions.get(a.session.id).state, 'live');
- assert.strictEqual(rt.store.workers.get(g1.id).state, 'draining');
+t('a draining generation keeps its live session; new sessions only reach the newest generation', async () => {
+ const { rt, coordinator, definition, key, worker } = await setup();
+ const g1 = await worker('rtmp-ingest');
+ const a = await rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker: g1 });
+ await rt.store.sessions.transition(a.session.id, 'live');
+ await worker('rtmp-ingest');
+ await coordinator.syncTick();
+ assert.strictEqual((await rt.store.sessions.get(a.session.id)).state, 'live');
+ assert.strictEqual((await rt.store.workers.get(g1.id)).state, 'draining');
// The worker runtime of the draining generation refuses to take sessions (rtmp-ingest checks
// The worker runtime of the draining generation refuses to take sessions (rtmp-ingest checks
// runtime.draining / state === 'ready' in its publish handler).
// runtime.draining / state === 'ready' in its publish handler).
const exits = [];
const exits = [];
const wr = createWorkerRuntime({ rt, kind: 'rtmp-ingest', log: silent, exit: (c) => exits.push(c), hooks: { activeCount: () => 1 } });
const wr = createWorkerRuntime({ rt, kind: 'rtmp-ingest', log: silent, exit: (c) => exits.push(c), hooks: { activeCount: () => 1 } });
- wr.register({});
- wr.ready();
+ await wr.register({});
+ await wr.ready();
wr.stop();
wr.stop();
});
});
-t('a worker without heartbeats is lost: its sessions fail (session.failed) and its outputs go back to pending', () => {
- const { rt, clock, coordinator, definition, key, worker } = setup();
- const ing = worker('rtmp-ingest');
- const rs = worker('restream');
- rt.store.outputs.createDestination(definition.id, { platform: 'custom', server_url: 'rtmp://127.0.0.1:1/app', stream_key: 'abcd1' });
- const a = rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker: ing });
- rt.store.sessions.transition(a.session.id, 'live');
- const tick1 = coordinator.syncTick();
+t('a worker without heartbeats is lost: its sessions fail (session.failed) and its outputs go back to pending', async () => {
+ const { rt, clock, coordinator, definition, key, worker } = await setup();
+ const ing = await worker('rtmp-ingest');
+ const rs = await worker('restream');
+ await rt.store.outputs.createDestination(definition.id, { platform: 'custom', server_url: 'rtmp://127.0.0.1:1/app', stream_key: 'abcd1' });
+ const a = await rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker: ing });
+ await rt.store.sessions.transition(a.session.id, 'live');
+ const tick1 = await coordinator.syncTick();
assert.strictEqual(tick1.outputsCreated, 1);
assert.strictEqual(tick1.outputsCreated, 1);
assert.strictEqual(tick1.outputsAssigned, 1);
assert.strictEqual(tick1.outputsAssigned, 1);
- const out = rt.store.outputs.outputsOfSession(a.session.id)[0];
+ const out = (await rt.store.outputs.outputsOfSession(a.session.id))[0];
assert.strictEqual(out.worker.id, rs.id);
assert.strictEqual(out.worker.id, rs.id);
// The restream worker dies; the ingest keeps heartbeating.
// The restream worker dies; the ingest keeps heartbeating.
clock.advance(rt.config.workers.leaseMs + 1000);
clock.advance(rt.config.workers.leaseMs + 1000);
- rt.store.workers.heartbeat(ing.id);
- const tick2 = coordinator.syncTick();
+ await rt.store.workers.heartbeat(ing.id);
+ const tick2 = await coordinator.syncTick();
assert.strictEqual(tick2.lost, 1);
assert.strictEqual(tick2.lost, 1);
- assert.strictEqual(rt.store.workers.get(rs.id).state, 'lost');
- assert.strictEqual(rt.store.sessions.get(a.session.id).state, 'live', 'a lost restream worker never ends the source session');
- assert.strictEqual(rt.store.outputs.getOutput(out.id).state, 'pending');
- assert.strictEqual(rt.store.outputs.getOutput(out.id).worker, null);
+ assert.strictEqual((await rt.store.workers.get(rs.id)).state, 'lost');
+ assert.strictEqual((await rt.store.sessions.get(a.session.id)).state, 'live', 'a lost restream worker never ends the source session');
+ assert.strictEqual((await rt.store.outputs.getOutput(out.id)).state, 'pending');
+ assert.strictEqual((await rt.store.outputs.getOutput(out.id)).worker, null);
// The lost worker waking up and stopping its ffmpeg must not overwrite the released output.
// The lost worker waking up and stopping its ffmpeg must not overwrite the released output.
- assert.strictEqual(rt.store.outputs.report(out.id, { state: 'stopped' }, { workerId: rs.id }), null);
- assert.strictEqual(rt.store.outputs.getOutput(out.id).state, 'pending');
+ assert.strictEqual(await rt.store.outputs.report(out.id, { state: 'stopped' }, { workerId: rs.id }), null);
+ assert.strictEqual((await rt.store.outputs.getOutput(out.id)).state, 'pending');
// A new restream generation picks it up.
// A new restream generation picks it up.
- const rs2 = worker('restream');
- assert.strictEqual(coordinator.syncTick().outputsAssigned, 1);
- assert.strictEqual(rt.store.outputs.getOutput(out.id).worker.id, rs2.id);
+ const rs2 = await worker('restream');
+ assert.strictEqual((await coordinator.syncTick()).outputsAssigned, 1);
+ assert.strictEqual((await rt.store.outputs.getOutput(out.id)).worker.id, rs2.id);
// Now the ingest worker dies too: its session fails.
// Now the ingest worker dies too: its session fails.
clock.advance(rt.config.workers.leaseMs + 1000);
clock.advance(rt.config.workers.leaseMs + 1000);
- rt.store.workers.heartbeat(rs2.id);
- coordinator.syncTick();
- assert.strictEqual(rt.store.sessions.get(a.session.id).state, 'failed');
- assert.strictEqual(rt.store.sessions.get(a.session.id).failure_reason, 'worker_lost');
- assert.ok(outboxTypes(rt.db).includes('openre.session.failed'));
- assert.strictEqual(rt.store.outputs.getOutput(out.id).desired, 'stop');
+ await rt.store.workers.heartbeat(rs2.id);
+ await coordinator.syncTick();
+ assert.strictEqual((await rt.store.sessions.get(a.session.id)).state, 'failed');
+ assert.strictEqual((await rt.store.sessions.get(a.session.id)).failure_reason, 'worker_lost');
+ assert.ok((await outboxTypes(rt.db)).includes('openre.session.failed'));
+ assert.strictEqual((await rt.store.outputs.getOutput(out.id)).desired, 'stop');
});
});
-t('outputs are only created for enabled, auto-start destinations with a key and no cooldown', () => {
- const { rt, clock, coordinator, definition, key, worker } = setup();
- const ing = worker('rtmp-ingest');
- worker('restream');
- rt.store.outputs.createDestination(definition.id, { platform: 'custom', server_url: 'rtmp://127.0.0.1:1/a', stream_key: 'k1111' });
- rt.store.outputs.createDestination(definition.id, { platform: 'custom', server_url: 'rtmp://127.0.0.1:1/b', stream_key: 'k2222', auto_start: false });
- rt.store.outputs.createDestination(definition.id, { platform: 'custom', server_url: 'rtmp://127.0.0.1:1/c', stream_key: 'k3333', enabled: false });
- const cooling = rt.store.outputs.createDestination(definition.id, { platform: 'custom', server_url: 'rtmp://127.0.0.1:1/d', stream_key: 'k4444' });
- rt.store.outputs.markDestinationFailure(cooling.id, 'dead');
- const a = rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker: ing });
- rt.store.sessions.transition(a.session.id, 'live');
- assert.strictEqual(coordinator.syncTick().outputsCreated, 1);
+t('outputs are only created for enabled, auto-start destinations with a key and no cooldown', async () => {
+ const { rt, clock, coordinator, definition, key, worker } = await setup();
+ const ing = await worker('rtmp-ingest');
+ await worker('restream');
+ await rt.store.outputs.createDestination(definition.id, { platform: 'custom', server_url: 'rtmp://127.0.0.1:1/a', stream_key: 'k1111' });
+ await rt.store.outputs.createDestination(definition.id, { platform: 'custom', server_url: 'rtmp://127.0.0.1:1/b', stream_key: 'k2222', auto_start: false });
+ await rt.store.outputs.createDestination(definition.id, { platform: 'custom', server_url: 'rtmp://127.0.0.1:1/c', stream_key: 'k3333', enabled: false });
+ const cooling = await rt.store.outputs.createDestination(definition.id, { platform: 'custom', server_url: 'rtmp://127.0.0.1:1/d', stream_key: 'k4444' });
+ await rt.store.outputs.markDestinationFailure(cooling.id, 'dead');
+ const a = await rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker: ing });
+ await rt.store.sessions.transition(a.session.id, 'live');
+ assert.strictEqual((await coordinator.syncTick()).outputsCreated, 1);
// A manual start ignores the cooldown (like Live's /start).
// A manual start ignores the cooldown (like Live's /start).
- rt.store.outputs.startDestination(cooling.id);
- assert.strictEqual(rt.store.outputs.outputsOfSession(a.session.id).length, 2);
+ await rt.store.outputs.startDestination(cooling.id);
+ assert.strictEqual((await rt.store.outputs.outputsOfSession(a.session.id)).length, 2);
clock.advance(1);
clock.advance(1);
});
});
-t('a restream generation at its drain deadline hands its outputs to the newest generation', () => {
- const { rt, coordinator, definition, key, worker } = setup();
- const ing = worker('rtmp-ingest');
- const rs1 = worker('restream');
- rt.store.outputs.createDestination(definition.id, { platform: 'custom', server_url: 'rtmp://127.0.0.1:1/a', stream_key: 'k1111' });
- const a = rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker: ing });
- rt.store.sessions.transition(a.session.id, 'live');
- coordinator.syncTick();
- const out = rt.store.outputs.outputsOfSession(a.session.id)[0];
- rt.store.outputs.report(out.id, { state: 'live' });
- const rs2 = worker('restream');
- coordinator.syncTick();
- assert.strictEqual(rt.store.workers.get(rs1.id).state, 'draining');
- assert.strictEqual(rt.store.outputs.getOutput(out.id).worker.id, rs1.id, 'the running output stays on the draining generation');
- assert.strictEqual(rt.store.outputs.release(rs1.id), 1); // what restream-worker does at its deadline
- coordinator.syncTick();
- assert.strictEqual(rt.store.outputs.getOutput(out.id).worker.id, rs2.id);
- assert.strictEqual(rt.store.sessions.get(a.session.id).state, 'live');
+t('a restream generation at its drain deadline hands its outputs to the newest generation', async () => {
+ const { rt, coordinator, definition, key, worker } = await setup();
+ const ing = await worker('rtmp-ingest');
+ const rs1 = await worker('restream');
+ await rt.store.outputs.createDestination(definition.id, { platform: 'custom', server_url: 'rtmp://127.0.0.1:1/a', stream_key: 'k1111' });
+ const a = await rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker: ing });
+ await rt.store.sessions.transition(a.session.id, 'live');
+ await coordinator.syncTick();
+ const out = (await rt.store.outputs.outputsOfSession(a.session.id))[0];
+ await rt.store.outputs.report(out.id, { state: 'live' });
+ const rs2 = await worker('restream');
+ await coordinator.syncTick();
+ assert.strictEqual((await rt.store.workers.get(rs1.id)).state, 'draining');
+ assert.strictEqual((await rt.store.outputs.getOutput(out.id)).worker.id, rs1.id, 'the running output stays on the draining generation');
+ assert.strictEqual(await rt.store.outputs.release(rs1.id), 1); // what restream-worker does at its deadline
+ await coordinator.syncTick();
+ assert.strictEqual((await rt.store.outputs.getOutput(out.id)).worker.id, rs2.id);
+ assert.strictEqual((await rt.store.sessions.get(a.session.id)).state, 'live');
});
});
t('only one coordinator acts at a time', async () => {
t('only one coordinator acts at a time', async () => {
- const { rt } = setup();
+ const { rt } = await setup();
const a = createCoordinator({ rt, media: null, log: silent, holder: 'a' });
const a = createCoordinator({ rt, media: null, log: silent, holder: 'a' });
const b = createCoordinator({ rt, media: null, log: silent, holder: 'b' });
const b = createCoordinator({ rt, media: null, log: silent, holder: 'b' });
assert.ok(!(await a.tick()).skipped);
assert.ok(!(await a.tick()).skipped);
@@ -141,29 +141,29 @@ t('only one coordinator acts at a time', async () => {
@@ -141,29 +141,29 @@ t('only one coordinator acts at a time', async () => {
});
});
t('the worker runtime exits by itself when a draining generation is idle, and on being lost', async () => {
t('the worker runtime exits by itself when a draining generation is idle, and on being lost', async () => {
- const { rt, worker } = setup();
+ const { rt, worker } = await setup();
const exits = [];
const exits = [];
let active = 1;
let active = 1;
const wr = createWorkerRuntime({ rt, kind: 'restream', log: silent, exit: (c) => exits.push(c), hooks: { activeCount: () => active } });
const wr = createWorkerRuntime({ rt, kind: 'restream', log: silent, exit: (c) => exits.push(c), hooks: { activeCount: () => active } });
- wr.register({});
- wr.ready();
- worker('restream');
- rt.store.workers.drain(wr.me.id, Date.now() + 60000);
- wr.beat();
+ await wr.register({});
+ await wr.ready();
+ await worker('restream');
+ await rt.store.workers.drain(wr.me.id, Date.now() + 60000);
+ await wr.beat();
assert.ok(wr.draining);
assert.ok(wr.draining);
assert.deepStrictEqual(exits, [], 'still carrying an output');
assert.deepStrictEqual(exits, [], 'still carrying an output');
active = 0;
active = 0;
- wr.beat();
+ await wr.beat();
await new Promise(r => setImmediate(r));
await new Promise(r => setImmediate(r));
assert.deepStrictEqual(exits, [0]);
assert.deepStrictEqual(exits, [0]);
- assert.strictEqual(rt.store.workers.get(wr.me.id).state, 'stopped');
+ assert.strictEqual((await rt.store.workers.get(wr.me.id)).state, 'stopped');
const exits2 = [];
const exits2 = [];
const wr2 = createWorkerRuntime({ rt, kind: 'rtmp-ingest', log: silent, exit: (c) => exits2.push(c), hooks: { activeCount: () => 1 } });
const wr2 = createWorkerRuntime({ rt, kind: 'rtmp-ingest', log: silent, exit: (c) => exits2.push(c), hooks: { activeCount: () => 1 } });
- wr2.register({});
- wr2.ready();
- rt.store.workers.lose(wr2.me.id, 'test');
- wr2.beat();
+ await wr2.register({});
+ await wr2.ready();
+ await rt.store.workers.lose(wr2.me.id, 'test');
+ await wr2.beat();
await new Promise(r => setImmediate(r));
await new Promise(r => setImmediate(r));
assert.deepStrictEqual(exits2, [1]);
assert.deepStrictEqual(exits2, [1]);
});
});
@@ -72,8 +72,8 @@ t('forms need the CSRF token; creating a stream shows the key once', async () =>
@@ -72,8 +72,8 @@ t('forms need the CSRF token; creating a stream shows the key once', async () =>
assert.strictEqual(created.status, 200);
assert.strictEqual(created.status, 200);
assert.strictEqual(created.headers.get('cache-control'), 'no-store');
assert.strictEqual(created.headers.get('cache-control'), 'no-store');
const key = /(ork_[A-Za-z0-9_-]{43})/.exec(created.text)[1];
const key = /(ork_[A-Za-z0-9_-]{43})/.exec(created.text)[1];
- const def = api.rt.store.definitions.list({ owner_subject: OWNER })[0];
- assert.strictEqual(api.rt.store.definitions.resolveIngestKey(key, 'rtmp').definition.id, def.id);
+ const def = (await api.rt.store.definitions.list({ owner_subject: OWNER }))[0];
+ assert.strictEqual((await api.rt.store.definitions.resolveIngestKey(key, 'rtmp')).definition.id, def.id);
const page = await request(api.base, 'GET', `/streams/${def.id}`, { cookie });
const page = await request(api.base, 'GET', `/streams/${def.id}`, { cookie });
assert.strictEqual(page.status, 200);
assert.strictEqual(page.status, 200);
assert.ok(!page.text.includes(key), 'the stream page never shows the key again');
assert.ok(!page.text.includes(key), 'the stream page never shows the key again');
@@ -91,7 +91,7 @@ t('forms need the CSRF token; creating a stream shows the key once', async () =>
@@ -91,7 +91,7 @@ t('forms need the CSRF token; creating a stream shows the key once', async () =>
const rotated = await request(api.base, 'POST', `/streams/${def.id}/rotate`, { cookie, form: { grace_seconds: '0', _csrf: csrfFor(token) } });
const rotated = await request(api.base, 'POST', `/streams/${def.id}/rotate`, { cookie, form: { grace_seconds: '0', _csrf: csrfFor(token) } });
const key2 = /(ork_[A-Za-z0-9_-]{43})/.exec(rotated.text)[1];
const key2 = /(ork_[A-Za-z0-9_-]{43})/.exec(rotated.text)[1];
assert.notStrictEqual(key2, key);
assert.notStrictEqual(key2, key);
- assert.strictEqual(api.rt.store.definitions.resolveIngestKey(key, 'rtmp').error, 'revoked_key');
+ assert.strictEqual((await api.rt.store.definitions.resolveIngestKey(key, 'rtmp')).error, 'revoked_key');
await api.close();
await api.close();
});
});
@@ -13,7 +13,7 @@ const { createRtmpIngest } = require('../workers/rtmp-ingest');
@@ -13,7 +13,7 @@ const { createRtmpIngest } = require('../workers/rtmp-ingest');
const { createRestreamWorker } = require('../workers/restream-worker');
const { createRestreamWorker } = require('../workers/restream-worker');
const { createUnportedTransport } = require('../workers/unported-transport');
const { createUnportedTransport } = require('../workers/unported-transport');
const { openDb } = require('../server/db');
const { openDb } = require('../server/db');
-const { runtime, bootApi, request, serviceToken, tmpDir, testEnv, child, suite, silent } = require('./helpers');
+const { runtime, bootApi, request, serviceToken, tmpDir, testEnv, child, suite, silent, testDb } = require('./helpers');
const t = suite('drill-mode');
const t = suite('drill-mode');
// What the production env file sets: with these, a non-drill process would publish and record.
// What the production env file sets: with these, a non-drill process would publish and record.
@@ -26,10 +26,10 @@ t('OPENRE_DRILL parses like the other switches and is off by default', () => {
@@ -26,10 +26,10 @@ t('OPENRE_DRILL parses like the other switches and is off by default', () => {
for (const v of ['', '0', 'false', 'off']) assert.strictEqual(load({ OPENRE_DRILL: v }).drill, false, v);
for (const v of ['', '0', 'false', 'off']) assert.strictEqual(load({ OPENRE_DRILL: v }).drill, false, v);
});
});
-t('no event relay and no Media client in a drill, even with production credentials', () => {
- const on = load(testEnv(tmpDir(), PROD_LIKE));
- const off = load(testEnv(tmpDir(), { ...PROD_LIKE, OPENRE_DRILL: '1' }));
- const db = openDb(':memory:');
+t('no event relay and no Media client in a drill, even with production credentials', async () => {
+ const on = load(await testEnv(tmpDir(), PROD_LIKE));
+ const off = load(await testEnv(tmpDir(), { ...PROD_LIKE, OPENRE_DRILL: '1' }));
+ const db = (await testDb(tmpDir())).db;
assert.strictEqual(createEvents({ config: on, db, log: silent }).configured, true, 'control: the same env publishes outside a drill');
assert.strictEqual(createEvents({ config: on, db, log: silent }).configured, true, 'control: the same env publishes outside a drill');
const ev = createEvents({ config: off, db, log: silent });
const ev = createEvents({ config: off, db, log: silent });
assert.strictEqual(ev.configured, false);
assert.strictEqual(ev.configured, false);
@@ -39,21 +39,21 @@ t('no event relay and no Media client in a drill, even with production credentia
@@ -39,21 +39,21 @@ t('no event relay and no Media client in a drill, even with production credentia
db.close();
db.close();
});
});
-t('the coordinator and every transport worker refuse to be created in a drill', () => {
- const rt = runtime({ env: { OPENRE_DRILL: '1' } });
+t('the coordinator and every transport worker refuse to be created in a drill', async () => {
+ const rt = await runtime({ env: { OPENRE_DRILL: '1' } });
const drill = (fn) => assert.throws(fn, (err) => err.code === 'OPENRE_DRILL' && /restore drill/.test(err.message));
const drill = (fn) => assert.throws(fn, (err) => err.code === 'OPENRE_DRILL' && /restore drill/.test(err.message));
drill(() => createCoordinator({ rt, media: null, log: silent }));
drill(() => createCoordinator({ rt, media: null, log: silent }));
drill(() => createRtmpIngest({ rt, log: silent, exit: () => {} }));
drill(() => createRtmpIngest({ rt, log: silent, exit: () => {} }));
drill(() => createRestreamWorker({ rt, log: silent, exit: () => {} }));
drill(() => createRestreamWorker({ rt, log: silent, exit: () => {} }));
for (const kind of ['webrtc-ingest', 'sfu', 'jsmpeg']) drill(() => createUnportedTransport({ rt, kind, log: silent, exit: () => {} }));
for (const kind of ['webrtc-ingest', 'sfu', 'jsmpeg']) drill(() => createUnportedTransport({ rt, kind, log: silent, exit: () => {} }));
- assert.strictEqual(rt.db.prepare('SELECT count(*) AS n FROM workers').get().n, 0, 'no generation registered');
- assert.strictEqual(rt.db.prepare('SELECT count(*) AS n FROM leases').get().n, 0, 'no coordinator lease taken');
+ assert.strictEqual((await rt.db.prepare('SELECT count(*) AS n FROM workers').get()).n, 0, 'no generation registered');
+ assert.strictEqual((await rt.db.prepare('SELECT count(*) AS n FROM leases').get()).n, 0, 'no coordinator lease taken');
});
});
t('the entry points exit before opening the database', async () => {
t('the entry points exit before opening the database', async () => {
for (const script of ['workers/coordinator.js', 'workers/rtmp-ingest.js', 'workers/restream-worker.js', 'workers/sfu.js']) {
for (const script of ['workers/coordinator.js', 'workers/rtmp-ingest.js', 'workers/restream-worker.js', 'workers/sfu.js']) {
const dir = tmpDir();
const dir = tmpDir();
- const p = child(script, testEnv(dir, { ...PROD_LIKE, OPENRE_DRILL: '1' }));
+ const p = child(script, await testEnv(dir, { ...PROD_LIKE, OPENRE_DRILL: '1' }));
const r = await p.exited;
const r = await p.exited;
assert.strictEqual(r.code, 1, `${script} exit code`);
assert.strictEqual(r.code, 1, `${script} exit code`);
assert.match(p.output, /does not run in a restore drill/, script);
assert.match(p.output, /does not run in a restore drill/, script);
@@ -73,7 +73,7 @@ t('openre-api in a drill: reads work and match a normal instance; writes, /play/
@@ -73,7 +73,7 @@ t('openre-api in a drill: reads work and match a normal instance; writes, /play/
assert.strictEqual(met.status, 200, 'loopback /metrics (Track O)');
assert.strictEqual(met.status, 200, 'loopback /metrics (Track O)');
const health = await request(normal.base, 'GET', '/api/health');
const health = await request(normal.base, 'GET', '/api/health');
const list = await request(normal.base, 'GET', '/api/v1/streams', { token: reader });
const list = await request(normal.base, 'GET', '/api/v1/streams', { token: reader });
- const outboxBefore = normal.rt.db.prepare('SELECT count(*) AS n FROM event_outbox').get().n;
+ const outboxBefore = (await normal.rt.db.prepare('SELECT count(*) AS n FROM event_outbox').get()).n;
await normal.close();
await normal.close();
const drill = await bootApi({ dir, env: { ...PROD_LIKE, OPENRE_DRILL: '1' } });
const drill = await bootApi({ dir, env: { ...PROD_LIKE, OPENRE_DRILL: '1' } });
@@ -100,9 +100,9 @@ t('openre-api in a drill: reads work and match a normal instance; writes, /play/
@@ -100,9 +100,9 @@ t('openre-api in a drill: reads work and match a normal instance; writes, /play/
assert.strictEqual(play.status, 503);
assert.strictEqual(play.status, 503);
const login = await request(drill.base, 'GET', '/auth/login');
const login = await request(drill.base, 'GET', '/auth/login');
assert.strictEqual(login.status, 503);
assert.strictEqual(login.status, 503);
- const count = drill.rt.db.prepare('SELECT count(*) AS n FROM stream_definitions').get().n;
+ const count = (await drill.rt.db.prepare('SELECT count(*) AS n FROM stream_definitions').get()).n;
assert.strictEqual(count, 1, 'nothing was written');
assert.strictEqual(count, 1, 'nothing was written');
- assert.strictEqual(drill.rt.db.prepare('SELECT count(*) AS n FROM event_outbox').get().n, outboxBefore, 'no event enqueued');
+ assert.strictEqual((await drill.rt.db.prepare('SELECT count(*) AS n FROM event_outbox').get()).n, outboxBefore, 'no event enqueued');
} finally {
} finally {
await drill.close();
await drill.close();
}
}
@@ -9,12 +9,14 @@ const { load } = require('../../server/config');
@@ -9,12 +9,14 @@ const { load } = require('../../server/config');
const { openRuntime } = require('../../server/store');
const { openRuntime } = require('../../server/store');
const { createWorkerRuntime } = require('../../workers/runtime');
const { createWorkerRuntime } = require('../../workers/runtime');
-const rt = openRuntime({ config: load(), log: { log() {}, warn() {}, error: console.error } });
-const worker = createWorkerRuntime({ rt, kind: 'rtmp-ingest', hooks: { activeCount: () => 1 }, log: { log() {}, warn() {}, error: console.error } });
-worker.register({ publicPort: 0, rtmpPlayPort: 0, flvPort: 0 });
-worker.ready();
-const definition = rt.store.definitions.get(process.env.FAKE_STREAM_ID);
-const a = rt.store.sessions.admit({ definition, key: null, protocol: 'rtmp', worker: worker.me });
-rt.store.sessions.transition(a.session.id, 'live', { reason: 'fake_media' });
-console.log(`SESSION ${a.session.id} WORKER ${worker.me.id}`);
-setInterval(() => {}, 1 << 30);
+(async () => {
+ const rt = await openRuntime({ config: load(), log: { log() {}, warn() {}, error: console.error } });
+ const worker = createWorkerRuntime({ rt, kind: 'rtmp-ingest', hooks: { activeCount: () => 1 }, log: { log() {}, warn() {}, error: console.error } });
+ await worker.register({ publicPort: 0, rtmpPlayPort: 0, flvPort: 0 });
+ await worker.ready();
+ const definition = await rt.store.definitions.get(process.env.FAKE_STREAM_ID);
+ const a = await rt.store.sessions.admit({ definition, key: null, protocol: 'rtmp', worker: worker.me });
+ await rt.store.sessions.transition(a.session.id, 'live', { reason: 'fake_media' });
+ console.log(`SESSION ${a.session.id} WORKER ${worker.me.id}`);
+ setInterval(() => {}, 1 << 30);
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -2,6 +2,10 @@
@@ -2,6 +2,10 @@
/**
/**
* Shared test fixtures: a generated Network signing key and token minting, temp databases, a
* Shared test fixtures: a generated Network signing key and token minting, temp databases, a
* manual clock, runtime/API boot helpers, child-process helpers and a tiny sequential runner.
* manual clock, runtime/API boot helpers, child-process helpers and a tiny sequential runner.
+ *
+ * Databases (ADR-035): one per test directory, migrated. npm test: PGlite in memory, so one process (a test that
+ * spawns OpenRe processes sharing it says it is skipped); npm run test:pg (OPENRE_TEST_STORE=pg): a schema of its own
+ * on the PostgreSQL + PgBouncer containers, whose URLs every spawned process of the directory gets.
*/
*/
const crypto = require('crypto');
const crypto = require('crypto');
const fs = require('fs');
const fs = require('fs');
@@ -10,8 +14,10 @@ const os = require('os');
@@ -10,8 +14,10 @@ const os = require('os');
const path = require('path');
const path = require('path');
const { spawn, spawnSync } = require('child_process');
const { spawn, spawnSync } = require('child_process');
const { serviceAuth } = require('openvibe-contracts');
const { serviceAuth } = require('openvibe-contracts');
+const { createTestDb } = require('openvibe-sdk/testing');
const { load } = require('../server/config');
const { load } = require('../server/config');
const { openRuntime } = require('../server/store');
const { openRuntime } = require('../server/store');
+const { MIGRATIONS } = require('../server/db');
const ROOT = path.join(__dirname, '..');
const ROOT = path.join(__dirname, '..');
const ISSUER = 'https://openvibe.network';
const ISSUER = 'https://openvibe.network';
@@ -53,10 +59,27 @@ function tmpDir() {
@@ -53,10 +59,27 @@ function tmpDir() {
return d;
return d;
}
}
-function testEnv(dir, extra = {}) {
+const STORE = process.env.OPENRE_TEST_STORE === 'pg' ? 'pg' : 'pglite';
+const databases = new Map();
+
+/** The migrated database of a test directory; every runtime, API and spawned process of the directory shares it. */
+async function testDb(dir) {
+ if (!databases.has(dir)) databases.set(dir, createTestDb({ migrations: MIGRATIONS, store: STORE, service: 'openre' }));
+ return await databases.get(dir);
+}
+
+/** Several OpenRe processes on one database need PostgreSQL (PGlite is one process). */
+const multiProcess = () => STORE === 'pg';
+
+/** A handle the test keeps: a runtime's or an API's close() leaves it open for the next one on the directory. */
+const keepOpen = (db) => new Proxy(db, { get: (t, k) => (k === 'close' ? async () => {} : typeof t[k] === 'function' ? t[k].bind(t) : t[k]) });
+
+async function testEnv(dir, extra = {}) {
+ const t = await testDb(dir);
return {
return {
NODE_ENV: 'test',
NODE_ENV: 'test',
PORT: '0',
PORT: '0',
+ ...(t.url ? { DATABASE_URL: t.url, DATABASE_DIRECT_URL: t.directUrl } : { OPENRE_PGLITE_DIR: path.join(dir, 'pglite') }),
OPENRE_DB_PATH: path.join(dir, 'openre.db'),
OPENRE_DB_PATH: path.join(dir, 'openre.db'),
OPENRE_SECRETS_KEY: SECRETS_KEY,
OPENRE_SECRETS_KEY: SECRETS_KEY,
OV_NETWORK_PUBLIC_KEY: publicKey,
OV_NETWORK_PUBLIC_KEY: publicKey,
@@ -67,15 +90,15 @@ function testEnv(dir, extra = {}) {
@@ -67,15 +90,15 @@ function testEnv(dir, extra = {}) {
}
}
/** A runtime on a temp database (no network). */
/** A runtime on a temp database (no network). */
-function runtime({ env = {}, clock, dir = tmpDir(), fetchImpl } = {}) {
- const config = load(testEnv(dir, env));
- return { ...openRuntime({ config, clock, log: silent, fetchImpl }), dir };
+async function runtime({ env = {}, clock, dir = tmpDir(), fetchImpl } = {}) {
+ const config = load(await testEnv(dir, env));
+ return { ...await openRuntime({ config, clock, log: silent, fetchImpl, db: keepOpen((await testDb(dir)).db) }), dir };
}
}
async function bootApi({ env = {}, dir = tmpDir(), clock, fetchImpl } = {}) {
async function bootApi({ env = {}, dir = tmpDir(), clock, fetchImpl } = {}) {
const { start } = require('../server/index');
const { start } = require('../server/index');
- const config = load(testEnv(dir, env));
- const h = await start({ config, clock, log: silent, fetchImpl });
+ const config = load(await testEnv(dir, env));
+ const h = await start({ config, clock, log: silent, fetchImpl, db: keepOpen((await testDb(dir)).db) });
const base = `http://127.0.0.1:${h.server.address().port}`;
const base = `http://127.0.0.1:${h.server.address().port}`;
return { ...h, base, dir };
return { ...h, base, dir };
}
}
@@ -173,14 +196,14 @@ function suite(name) {
@@ -173,14 +196,14 @@ function suite(name) {
}
}
/** The event types in the outbox, in order. */
/** The event types in the outbox, in order. */
-function outboxTypes(db) {
- return db.prepare('SELECT envelope FROM event_outbox ORDER BY id').all().map(r => JSON.parse(r.envelope).event_type);
+async function outboxTypes(db) {
+ return (await db.prepare('SELECT envelope FROM event_outbox ORDER BY id').all()).map(r => (typeof r.envelope === 'string' ? JSON.parse(r.envelope) : r.envelope).event_type);
}
}
-function outboxEnvelopes(db) {
- return db.prepare('SELECT envelope FROM event_outbox ORDER BY id').all().map(r => JSON.parse(r.envelope));
+async function outboxEnvelopes(db) {
+ return (await db.prepare('SELECT envelope FROM event_outbox ORDER BY id').all()).map(r => (typeof r.envelope === 'string' ? JSON.parse(r.envelope) : r.envelope));
}
}
module.exports = {
module.exports = {
- ROOT, ISSUER, privateKey, publicKey, SECRETS_KEY, OWNER, OTHER, silent, serviceToken, userToken, manualClock, tmpDir, testEnv,
+ ROOT, ISSUER, privateKey, publicKey, SECRETS_KEY, OWNER, OTHER, silent, serviceToken, userToken, manualClock, tmpDir, testEnv, testDb, multiProcess, keepOpen,
runtime, bootApi, request, freePort, child, waitFor, sleep, hasFfmpeg, publish, rtmpSink, suite, outboxTypes, outboxEnvelopes,
runtime, bootApi, request, freePort, child, waitFor, sleep, hasFfmpeg, publish, rtmpSink, suite, outboxTypes, outboxEnvelopes,
};
};
@@ -8,38 +8,38 @@ const { createCoordinator } = require('../server/coordinator');
@@ -8,38 +8,38 @@ const { createCoordinator } = require('../server/coordinator');
const t = suite('keys');
const t = suite('keys');
-t('a new definition issues one key; only its SHA-256 is stored', () => {
- const rt = runtime();
- const { definition, key } = rt.store.definitions.create({ owner_subject: OWNER, title: 'Main' });
+t('a new definition issues one key; only its SHA-256 is stored', async () => {
+ const rt = await runtime();
+ const { definition, key } = await rt.store.definitions.create({ owner_subject: OWNER, title: 'Main' });
assert.ok(isIngestKeyShape(key.key), 'ork_ + 43 base64url chars');
assert.ok(isIngestKeyShape(key.key), 'ork_ + 43 base64url chars');
- const rows = rt.db.prepare('SELECT * FROM ingest_keys').all();
+ const rows = await rt.db.prepare('SELECT * FROM ingest_keys').all();
assert.strictEqual(rows.length, 1);
assert.strictEqual(rows.length, 1);
assert.strictEqual(rows[0].key_hash, hashIngestKey(key.key));
assert.strictEqual(rows[0].key_hash, hashIngestKey(key.key));
assert.strictEqual(rows[0].hint, key.key.slice(-4));
assert.strictEqual(rows[0].hint, key.key.slice(-4));
// The plain key is nowhere in the database file's tables.
// The plain key is nowhere in the database file's tables.
- const dump = JSON.stringify(rt.db.prepare("SELECT * FROM ingest_keys").all()) + JSON.stringify(rt.db.prepare('SELECT * FROM stream_definitions').all());
+ const dump = JSON.stringify(await rt.db.prepare("SELECT * FROM ingest_keys").all()) + JSON.stringify(await rt.db.prepare('SELECT * FROM stream_definitions').all());
assert.ok(!dump.includes(key.key));
assert.ok(!dump.includes(key.key));
- const r = rt.store.definitions.resolveIngestKey(key.key, 'rtmp');
+ const r = await rt.store.definitions.resolveIngestKey(key.key, 'rtmp');
assert.strictEqual(r.definition.id, definition.id);
assert.strictEqual(r.definition.id, definition.id);
- assert.ok(rt.db.prepare('SELECT last_used_at FROM ingest_keys').get().last_used_at > 0);
+ assert.ok((await rt.db.prepare('SELECT last_used_at FROM ingest_keys').get()).last_used_at > 0);
});
});
-t('unknown, malformed and wrong-protocol keys are refused', () => {
- const rt = runtime();
- const { key } = rt.store.definitions.create({ owner_subject: OWNER });
- assert.strictEqual(rt.store.definitions.resolveIngestKey('ork_' + 'A'.repeat(43), 'rtmp').error, 'unknown_key');
- assert.strictEqual(rt.store.definitions.resolveIngestKey('a3f9c2', 'rtmp').error, 'malformed_key');
- assert.strictEqual(rt.store.definitions.resolveIngestKey(key.key, 'whip').error, 'protocol_not_allowed');
+t('unknown, malformed and wrong-protocol keys are refused', async () => {
+ const rt = await runtime();
+ const { key } = await rt.store.definitions.create({ owner_subject: OWNER });
+ assert.strictEqual((await rt.store.definitions.resolveIngestKey('ork_' + 'A'.repeat(43), 'rtmp')).error, 'unknown_key');
+ assert.strictEqual((await rt.store.definitions.resolveIngestKey('a3f9c2', 'rtmp')).error, 'malformed_key');
+ assert.strictEqual((await rt.store.definitions.resolveIngestKey(key.key, 'whip')).error, 'protocol_not_allowed');
});
});
-t('rotate revokes the old key immediately and emits openre.key.rotated without any key', () => {
- const rt = runtime();
- const { definition, key: first } = rt.store.definitions.create({ owner_subject: OWNER, external_refs: [{ service: 'live', type: 'managed_stream', id: '7' }] });
- const r = rt.store.definitions.rotateKey(definition.id, { rotated_by: OWNER });
+t('rotate revokes the old key immediately and emits openre.key.rotated without any key', async () => {
+ const rt = await runtime();
+ const { definition, key: first } = await rt.store.definitions.create({ owner_subject: OWNER, external_refs: [{ service: 'live', type: 'managed_stream', id: '7' }] });
+ const r = await rt.store.definitions.rotateKey(definition.id, { rotated_by: OWNER });
assert.notStrictEqual(r.key.key, first.key);
assert.notStrictEqual(r.key.key, first.key);
- assert.strictEqual(rt.store.definitions.resolveIngestKey(first.key, 'rtmp').error, 'revoked_key');
- assert.strictEqual(rt.store.definitions.resolveIngestKey(r.key.key, 'rtmp').definition.id, definition.id);
- const ev = outboxEnvelopes(rt.db).find(e => e.event_type === 'openre.key.rotated');
+ assert.strictEqual((await rt.store.definitions.resolveIngestKey(first.key, 'rtmp')).error, 'revoked_key');
+ assert.strictEqual((await rt.store.definitions.resolveIngestKey(r.key.key, 'rtmp')).definition.id, definition.id);
+ const ev = (await outboxEnvelopes(rt.db)).find(e => e.event_type === 'openre.key.rotated');
assert.ok(ev);
assert.ok(ev);
assert.deepStrictEqual(ev.subject, { type: 'stream', id: definition.id, revision: 1 });
assert.deepStrictEqual(ev.subject, { type: 'stream', id: definition.id, revision: 1 });
assert.deepStrictEqual(ev.actor, { type: 'user', id: OWNER });
assert.deepStrictEqual(ev.actor, { type: 'user', id: OWNER });
@@ -50,48 +50,48 @@ t('rotate revokes the old key immediately and emits openre.key.rotated without a
@@ -50,48 +50,48 @@ t('rotate revokes the old key immediately and emits openre.key.rotated without a
t('rotate with a grace period keeps the old key for new publishes until the coordinator revokes it', async () => {
t('rotate with a grace period keeps the old key for new publishes until the coordinator revokes it', async () => {
const clock = manualClock();
const clock = manualClock();
- const rt = runtime({ clock });
- const { definition, key: first } = rt.store.definitions.create({ owner_subject: OWNER });
- const r = rt.store.definitions.rotateKey(definition.id, { grace_seconds: 600 });
- assert.strictEqual(rt.store.definitions.resolveIngestKey(first.key, 'rtmp').key.status, 'grace');
- assert.ok(rt.store.definitions.resolveIngestKey(r.key.key, 'rtmp').definition);
+ const rt = await runtime({ clock });
+ const { definition, key: first } = await rt.store.definitions.create({ owner_subject: OWNER });
+ const r = await rt.store.definitions.rotateKey(definition.id, { grace_seconds: 600 });
+ assert.strictEqual((await rt.store.definitions.resolveIngestKey(first.key, 'rtmp')).key.status, 'grace');
+ assert.ok((await rt.store.definitions.resolveIngestKey(r.key.key, 'rtmp')).definition);
clock.advance(601 * 1000);
clock.advance(601 * 1000);
// Past the grace period the key no longer authenticates, even before the coordinator runs...
// Past the grace period the key no longer authenticates, even before the coordinator runs...
- assert.strictEqual(rt.store.definitions.resolveIngestKey(first.key, 'rtmp').error, 'revoked_key');
+ assert.strictEqual((await rt.store.definitions.resolveIngestKey(first.key, 'rtmp')).error, 'revoked_key');
// ...and the coordinator records the revocation.
// ...and the coordinator records the revocation.
- const out = createCoordinator({ rt, media: null, log: { log() {}, warn() {}, error() {} } }).syncTick();
+ const out = await createCoordinator({ rt, media: null, log: { log() {}, warn() {}, error() {} } }).syncTick();
assert.strictEqual(out.keysRevoked, 1);
assert.strictEqual(out.keysRevoked, 1);
- assert.strictEqual(rt.db.prepare("SELECT status FROM ingest_keys WHERE key_hash = ?").get(hashIngestKey(first.key)).status, 'revoked');
+ assert.strictEqual((await rt.db.prepare("SELECT status FROM ingest_keys WHERE key_hash = ?").get(hashIngestKey(first.key))).status, 'revoked');
});
});
-t('archiving revokes every key and is refused while a session is open', () => {
- const rt = runtime();
- const { definition, key } = rt.store.definitions.create({ owner_subject: OWNER });
- const w = rt.store.workers.register({ kind: 'rtmp-ingest' });
- rt.store.workers.ready(w.id);
- const a = rt.store.sessions.admit({ definition, key: { id: 'k' }, protocol: 'rtmp', worker: rt.store.workers.get(w.id) });
- assert.throws(() => rt.store.definitions.archive(definition.id), /end the live session/);
- rt.store.sessions.finish(a.session.id, { reason: 'test' });
- rt.store.definitions.archive(definition.id);
- assert.strictEqual(rt.store.definitions.resolveIngestKey(key.key, 'rtmp').error, 'revoked_key');
+t('archiving revokes every key and is refused while a session is open', async () => {
+ const rt = await runtime();
+ const { definition, key } = await rt.store.definitions.create({ owner_subject: OWNER });
+ const w = await rt.store.workers.register({ kind: 'rtmp-ingest' });
+ await rt.store.workers.ready(w.id);
+ const a = await rt.store.sessions.admit({ definition, key: { id: 'k' }, protocol: 'rtmp', worker: await rt.store.workers.get(w.id) });
+ await assert.rejects(async () => await rt.store.definitions.archive(definition.id), /end the live session/);
+ await rt.store.sessions.finish(a.session.id, { reason: 'test' });
+ await rt.store.definitions.archive(definition.id);
+ assert.strictEqual((await rt.store.definitions.resolveIngestKey(key.key, 'rtmp')).error, 'revoked_key');
});
});
-t('external refs are unique across definitions', () => {
- const rt = runtime();
- rt.store.definitions.create({ owner_subject: OWNER, external_refs: [{ service: 'live', type: 'managed_stream', id: '9' }] });
- assert.throws(() => rt.store.definitions.create({ owner_subject: OWNER, external_refs: [{ service: 'live', type: 'managed_stream', id: '9' }] }), /already belongs/);
- assert.ok(rt.store.definitions.findByRef('live', 'managed_stream', '9'));
+t('external refs are unique across definitions', async () => {
+ const rt = await runtime();
+ await rt.store.definitions.create({ owner_subject: OWNER, external_refs: [{ service: 'live', type: 'managed_stream', id: '9' }] });
+ await assert.rejects(async () => await rt.store.definitions.create({ owner_subject: OWNER, external_refs: [{ service: 'live', type: 'managed_stream', id: '9' }] }), /already belongs/);
+ assert.ok(await rt.store.definitions.findByRef('live', 'managed_stream', '9'));
});
});
-t('destination keys are sealed with AES-256-GCM and only the last four characters come back', () => {
- const rt = runtime();
- const { definition } = rt.store.definitions.create({ owner_subject: OWNER });
- const d = rt.store.outputs.createDestination(definition.id, { platform: 'twitch', server_url: 'rtmps://live.twitch.tv/app', stream_key: 'live_1234567890_secret' });
+t('destination keys are sealed with AES-256-GCM and only the last four characters come back', async () => {
+ const rt = await runtime();
+ const { definition } = await rt.store.definitions.create({ owner_subject: OWNER });
+ const d = await rt.store.outputs.createDestination(definition.id, { platform: 'twitch', server_url: 'rtmps://live.twitch.tv/app', stream_key: 'live_1234567890_secret' });
assert.strictEqual(d.stream_key_hint, '****cret');
assert.strictEqual(d.stream_key_hint, '****cret');
assert.ok(!JSON.stringify(d).includes('live_1234567890_secret'));
assert.ok(!JSON.stringify(d).includes('live_1234567890_secret'));
- const row = rt.db.prepare('SELECT stream_key_enc FROM destinations').get();
+ const row = await rt.db.prepare('SELECT stream_key_enc FROM destinations').get();
assert.ok(row.stream_key_enc.startsWith('v1.') && !row.stream_key_enc.includes('secret'));
assert.ok(row.stream_key_enc.startsWith('v1.') && !row.stream_key_enc.includes('secret'));
- assert.strictEqual(rt.store.outputs.destinationForWorker(d.id).stream_key, 'live_1234567890_secret');
+ assert.strictEqual((await rt.store.outputs.destinationForWorker(d.id)).stream_key, 'live_1234567890_secret');
// A second key can take over: the previous one still opens old rows.
// A second key can take over: the previous one still opens old rows.
const box2 = createBox({ key: require('crypto').randomBytes(32).toString('hex'), previous: rt.config.secretsKey });
const box2 = createBox({ key: require('crypto').randomBytes(32).toString('hex'), previous: rt.config.secretsKey });
assert.strictEqual(box2.open(row.stream_key_enc), 'live_1234567890_secret');
assert.strictEqual(box2.open(row.stream_key_enc), 'live_1234567890_secret');
@@ -8,9 +8,9 @@ const { createLineage } = require('../server/lineage');
@@ -8,9 +8,9 @@ const { createLineage } = require('../server/lineage');
const t = suite('lineage');
const t = suite('lineage');
-function setup(answer) {
+async function setup(answer) {
const clock = manualClock();
const clock = manualClock();
- const rt = runtime({ clock });
+ const rt = await runtime({ clock });
const asked = [];
const asked = [];
const fetchImpl = async (url, opts = {}) => {
const fetchImpl = async (url, opts = {}) => {
const u = new URL(url);
const u = new URL(url);
@@ -29,34 +29,34 @@ function setup(answer) {
@@ -29,34 +29,34 @@ function setup(answer) {
const RESOLVED = { status: 'resolved', channel: { id: '17', slug: 'goosely', owner_subject: OWNER }, rule: 'slot', confidence: 'exact' };
const RESOLVED = { status: 'resolved', channel: { id: '17', slug: 'goosely', owner_subject: OWNER }, rule: 'slot', confidence: 'exact' };
t('a Live-linked definition is resolved with its owner and slot; an unlinked one is never asked about', async () => {
t('a Live-linked definition is resolved with its owner and slot; an unlinked one is never asked about', async () => {
- const { rt, lineage, asked } = setup(RESOLVED);
- const linked = rt.store.definitions.create({ owner_subject: OWNER, mirror_to_live: true, external_refs: [{ service: 'live', type: 'managed_stream', id: '12' }] }).definition;
- rt.store.definitions.create({ owner_subject: OWNER, title: 'OpenRe only' });
+ const { rt, lineage, asked } = await setup(RESOLVED);
+ const linked = (await rt.store.definitions.create({ owner_subject: OWNER, mirror_to_live: true, external_refs: [{ service: 'live', type: 'managed_stream', id: '12' }] })).definition;
+ await rt.store.definitions.create({ owner_subject: OWNER, title: 'OpenRe only' });
const out = await lineage.refresh();
const out = await lineage.refresh();
assert.deepStrictEqual(out, { checked: 1, resolved: 1 });
assert.deepStrictEqual(out, { checked: 1, resolved: 1 });
assert.deepStrictEqual(asked, [{ owner_subject: OWNER, slot_id: '12' }], 'owner subject and slot id, never a display name');
assert.deepStrictEqual(asked, [{ owner_subject: OWNER, slot_id: '12' }], 'owner subject and slot id, never a display name');
- const row = rt.db.prepare('SELECT resolution FROM definition_lineage WHERE definition_id = ?').get(linked.id);
+ const row = await rt.db.prepare('SELECT resolution FROM definition_lineage WHERE definition_id = ?').get(linked.id);
assert.strictEqual(JSON.parse(row.resolution).channel.slug, 'goosely');
assert.strictEqual(JSON.parse(row.resolution).channel.slug, 'goosely');
assert.deepStrictEqual(await lineage.refresh(), { checked: 0, resolved: 0 }, 'fresh answers are not asked again');
assert.deepStrictEqual(await lineage.refresh(), { checked: 0, resolved: 0 }, 'fresh answers are not asked again');
});
});
t('session events carry the resolved channel', async () => {
t('session events carry the resolved channel', async () => {
- const { rt, coordinator } = setup(RESOLVED);
- const { definition, key } = rt.store.definitions.create({ owner_subject: OWNER, mirror_to_live: true, external_refs: [{ service: 'live', type: 'managed_stream', id: '12' }] });
+ const { rt, coordinator } = await setup(RESOLVED);
+ const { definition, key } = await rt.store.definitions.create({ owner_subject: OWNER, mirror_to_live: true, external_refs: [{ service: 'live', type: 'managed_stream', id: '12' }] });
await coordinator.tick();
await coordinator.tick();
- const w = rt.store.workers.register({ kind: 'rtmp-ingest', endpoints: { flvPort: 19361, rtmpPlayPort: 19360 } }); rt.store.workers.ready(w.id);
- const a = rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker: rt.store.workers.get(w.id) });
- rt.store.sessions.transition(a.session.id, 'live');
- const started = outboxEnvelopes(rt.db).find((e) => e.event_type === 'openre.session.started');
+ const w = await rt.store.workers.register({ kind: 'rtmp-ingest', endpoints: { flvPort: 19361, rtmpPlayPort: 19360 } }); await rt.store.workers.ready(w.id);
+ const a = await rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker: await rt.store.workers.get(w.id) });
+ await rt.store.sessions.transition(a.session.id, 'live');
+ const started = (await outboxEnvelopes(rt.db)).find((e) => e.event_type === 'openre.session.started');
assert.deepStrictEqual(started.payload.lineage, { channel: { service: 'live', id: '17', slug: 'goosely', owner_subject: OWNER }, rule: 'slot' });
assert.deepStrictEqual(started.payload.lineage, { channel: { service: 'live', id: '17', slug: 'goosely', owner_subject: OWNER }, rule: 'slot' });
});
});
t('unresolved answers are kept but not carried; a stale answer is asked again; Live down waits for the next tick', async () => {
t('unresolved answers are kept but not carried; a stale answer is asked again; Live down waits for the next tick', async () => {
let mode = 'unresolved';
let mode = 'unresolved';
- const { rt, clock, lineage } = setup((u) => (mode === 'down' ? { status: 503, ok: false, json: async () => ({}) } : { status: 200, ok: true, json: async () => (mode === 'unresolved' ? { status: 'unresolved', reason: 'no_match' } : RESOLVED) }));
- const { definition } = rt.store.definitions.create({ owner_subject: OWNER, mirror_to_live: true });
+ const { rt, clock, lineage } = await setup((u) => (mode === 'down' ? { status: 503, ok: false, json: async () => ({}) } : { status: 200, ok: true, json: async () => (mode === 'unresolved' ? { status: 'unresolved', reason: 'no_match' } : RESOLVED) }));
+ const { definition } = await rt.store.definitions.create({ owner_subject: OWNER, mirror_to_live: true });
assert.deepStrictEqual(await lineage.refresh(), { checked: 1, resolved: 0 });
assert.deepStrictEqual(await lineage.refresh(), { checked: 1, resolved: 0 });
- assert.strictEqual(rt.store.sessions.liveLineage(definition.id), null, 'an unresolved answer is not carried');
+ assert.strictEqual(await rt.store.sessions.liveLineage(definition.id), null, 'an unresolved answer is not carried');
mode = 'down';
mode = 'down';
clock.advance(16 * 60 * 1000);
clock.advance(16 * 60 * 1000);
assert.deepStrictEqual(await lineage.refresh(), { checked: 0, resolved: 0 });
assert.deepStrictEqual(await lineage.refresh(), { checked: 0, resolved: 0 });
@@ -67,7 +67,7 @@ t('unresolved answers are kept but not carried; a stale answer is asked again; L
@@ -67,7 +67,7 @@ t('unresolved answers are kept but not carried; a stale answer is asked again; L
t('off without a service secret', async () => {
t('off without a service secret', async () => {
const clock = manualClock();
const clock = manualClock();
- const rt = runtime({ clock });
+ const rt = await runtime({ clock });
const off = createLineage({ db: rt.db, config: { ...rt.config, oauth: { ...rt.config.oauth, clientSecret: '' } }, env: {}, fetchImpl: () => { throw new Error('no calls'); }, log: silent });
const off = createLineage({ db: rt.db, config: { ...rt.config, oauth: { ...rt.config.oauth, clientSecret: '' } }, env: {}, fetchImpl: () => { throw new Error('no calls'); }, log: silent });
assert.strictEqual(off.enabled, false);
assert.strictEqual(off.enabled, false);
assert.deepStrictEqual(await off.refresh(), { skipped: 'off' });
assert.deepStrictEqual(await off.refresh(), { skipped: 'off' });
@@ -45,13 +45,13 @@ function liveSnapshot(dir) {
@@ -45,13 +45,13 @@ function liveSnapshot(dir) {
return file;
return file;
}
}
-t('dry run writes nothing', () => {
+t('dry run writes nothing', async () => {
const dir = tmpDir();
const dir = tmpDir();
- const rt = runtime({ dir });
+ const rt = await runtime({ dir });
const live = new Database(liveSnapshot(dir), { readonly: true });
const live = new Database(liveSnapshot(dir), { readonly: true });
- const report = migrate({ liveDb: live, rt, apply: false });
- assert.strictEqual(rt.db.prepare('SELECT COUNT(*) AS n FROM stream_definitions').get().n, 0);
- assert.strictEqual(rt.db.prepare('SELECT COUNT(*) AS n FROM migration_map').get().n, 0);
+ const report = await migrate({ liveDb: live, rt, apply: false });
+ assert.strictEqual((await rt.db.prepare('SELECT COUNT(*) AS n FROM stream_definitions').get()).n, 0);
+ assert.strictEqual((await rt.db.prepare('SELECT COUNT(*) AS n FROM migration_map').get()).n, 0);
assert.ok(report.counts.imported > 0);
assert.ok(report.counts.imported > 0);
assert.match(checklist(report), /dry run/);
assert.match(checklist(report), /dry run/);
});
});
@@ -59,17 +59,17 @@ t('dry run writes nothing', () => {
@@ -59,17 +59,17 @@ t('dry run writes nothing', () => {
let dir;
let dir;
let rt;
let rt;
let liveFile;
let liveFile;
-t('apply: slots with a subject are imported with new keys; the rest are held or excluded with reasons', () => {
+t('apply: slots with a subject are imported with new keys; the rest are held or excluded with reasons', async () => {
dir = tmpDir();
dir = tmpDir();
- rt = runtime({ dir, env: { OPENRE_DEST_ALLOW_PRIVATE: '0' } });
+ rt = await runtime({ dir, env: { OPENRE_DEST_ALLOW_PRIVATE: '0' } });
liveFile = liveSnapshot(dir);
liveFile = liveSnapshot(dir);
const before = fs.readFileSync(liveFile);
const before = fs.readFileSync(liveFile);
const live = new Database(liveFile, { readonly: true });
const live = new Database(liveFile, { readonly: true });
- const report = migrate({ liveDb: live, rt, apply: true });
+ const report = await migrate({ liveDb: live, rt, apply: true });
live.close();
live.close();
assert.ok(before.equals(fs.readFileSync(liveFile)), 'the Live snapshot is untouched');
assert.ok(before.equals(fs.readFileSync(liveFile)), 'the Live snapshot is untouched');
- const map = Object.fromEntries(rt.db.prepare('SELECT source_type, source_id, status, reason, target_id FROM migration_map').all().map(r => [`${r.source_type}:${r.source_id}`, r]));
+ const map = Object.fromEntries((await rt.db.prepare('SELECT source_type, source_id, status, reason, target_id FROM migration_map').all()).map(r => [`${r.source_type}:${r.source_id}`, r]));
assert.strictEqual(map['managed_stream:10'].status, 'imported');
assert.strictEqual(map['managed_stream:10'].status, 'imported');
assert.strictEqual(map['managed_stream:11'].status, 'held');
assert.strictEqual(map['managed_stream:11'].status, 'held');
assert.match(map['managed_stream:11'].reason, /no canonical subject/);
assert.match(map['managed_stream:11'].reason, /no canonical subject/);
@@ -77,21 +77,21 @@ t('apply: slots with a subject are imported with new keys; the rest are held or
@@ -77,21 +77,21 @@ t('apply: slots with a subject are imported with new keys; the rest are held or
assert.strictEqual(map['managed_stream:13'].status, 'imported');
assert.strictEqual(map['managed_stream:13'].status, 'imported');
assert.strictEqual(map['managed_stream:14'].status, 'imported');
assert.strictEqual(map['managed_stream:14'].status, 'imported');
- const def10 = rt.store.definitions.get(map['managed_stream:10'].target_id);
+ const def10 = await rt.store.definitions.get(map['managed_stream:10'].target_id);
assert.strictEqual(def10.owner_subject, SUBJECT_A);
assert.strictEqual(def10.owner_subject, SUBJECT_A);
assert.strictEqual(def10.recording_visibility, 'unlisted', 'from the channel default');
assert.strictEqual(def10.recording_visibility, 'unlisted', 'from the channel default');
assert.strictEqual(def10.recording_mode, 'vod');
assert.strictEqual(def10.recording_mode, 'vod');
assert.deepStrictEqual(def10.external_refs.map(r => `${r.service}:${r.type}:${r.id}`), ['live:managed_stream:10', 'live:user:1']);
assert.deepStrictEqual(def10.external_refs.map(r => `${r.service}:${r.type}:${r.id}`), ['live:managed_stream:10', 'live:user:1']);
- assert.strictEqual(rt.store.definitions.get(map['managed_stream:13'].target_id).recording_mode, 'none', 'VOD off on the channel, clips off on the slot');
- assert.strictEqual(rt.store.definitions.get(map['managed_stream:14'].target_id).recording_mode, 'clips');
- assert.strictEqual(rt.store.definitions.get(map['managed_stream:14'].target_id).recording_visibility, 'private');
+ assert.strictEqual((await rt.store.definitions.get(map['managed_stream:13'].target_id)).recording_mode, 'none', 'VOD off on the channel, clips off on the slot');
+ assert.strictEqual((await rt.store.definitions.get(map['managed_stream:14'].target_id)).recording_mode, 'clips');
+ assert.strictEqual((await rt.store.definitions.get(map['managed_stream:14'].target_id)).recording_visibility, 'private');
// Old keys: none of them authenticates, and none is stored in any form.
// Old keys: none of them authenticates, and none is stored in any form.
for (const old of ['a3f9c2leakedkey0000000000000000000000000', 'personalkey111']) {
for (const old of ['a3f9c2leakedkey0000000000000000000000000', 'personalkey111']) {
- assert.strictEqual(rt.db.prepare('SELECT COUNT(*) AS n FROM ingest_keys WHERE key_hash = ?').get(hashIngestKey(old)).n, 0);
- assert.ok(rt.store.definitions.resolveIngestKey(old, 'rtmp').error);
+ assert.strictEqual((await rt.db.prepare('SELECT COUNT(*) AS n FROM ingest_keys WHERE key_hash = ?').get(hashIngestKey(old))).n, 0);
+ assert.ok((await rt.store.definitions.resolveIngestKey(old, 'rtmp')).error);
}
}
- assert.strictEqual(rt.db.prepare("SELECT COUNT(*) AS n FROM ingest_keys WHERE status = 'active'").get().n, 3, 'one new key per imported slot');
+ assert.strictEqual((await rt.db.prepare("SELECT COUNT(*) AS n FROM ingest_keys WHERE status = 'active'").get()).n, 3, 'one new key per imported slot');
// Destinations: sealed, LAN one held, OAuth-linked flagged, unbound one held (two slots).
// Destinations: sealed, LAN one held, OAuth-linked flagged, unbound one held (two slots).
assert.strictEqual(map['restream_destination:100'].status, 'imported');
assert.strictEqual(map['restream_destination:100'].status, 'imported');
@@ -100,13 +100,13 @@ t('apply: slots with a subject are imported with new keys; the rest are held or
@@ -100,13 +100,13 @@ t('apply: slots with a subject are imported with new keys; the rest are held or
assert.match(map['restream_destination:102'].reason, /private/);
assert.match(map['restream_destination:102'].reason, /private/);
assert.strictEqual(map['restream_destination:103'].status, 'held');
assert.strictEqual(map['restream_destination:103'].status, 'held');
assert.match(map['restream_destination:103'].reason, /several slots/);
assert.match(map['restream_destination:103'].reason, /several slots/);
- const d100 = rt.store.outputs.destinationForWorker(map['restream_destination:100'].target_id);
+ const d100 = await rt.store.outputs.destinationForWorker(map['restream_destination:100'].target_id);
assert.strictEqual(d100.stream_key, 'live_twitchsecret_1');
assert.strictEqual(d100.stream_key, 'live_twitchsecret_1');
- assert.ok(!JSON.stringify(rt.db.prepare('SELECT * FROM destinations').all()).includes('live_twitchsecret_1'), 'sealed at rest');
- const d101 = rt.store.outputs.getDestination(map['restream_destination:101'].target_id);
+ assert.ok(!JSON.stringify(await rt.db.prepare('SELECT * FROM destinations').all()).includes('live_twitchsecret_1'), 'sealed at rest');
+ const d101 = await rt.store.outputs.getDestination(map['restream_destination:101'].target_id);
assert.strictEqual(d101.auto_start, false);
assert.strictEqual(d101.auto_start, false);
assert.strictEqual(d101.custom_video_bitrate, 4500);
assert.strictEqual(d101.custom_video_bitrate, 4500);
- const d102 = rt.store.outputs.getDestination(map['restream_destination:102'].target_id);
+ const d102 = await rt.store.outputs.getDestination(map['restream_destination:102'].target_id);
assert.strictEqual(d102.enabled, false);
assert.strictEqual(d102.enabled, false);
assert.ok(d102.hold_reason);
assert.ok(d102.hold_reason);
@@ -118,32 +118,32 @@ t('apply: slots with a subject are imported with new keys; the rest are held or
@@ -118,32 +118,32 @@ t('apply: slots with a subject are imported with new keys; the rest are held or
for (const secret of ['a3f9c2leakedkey', 'live_twitchsecret_1', 'yt-secret-2222', 'personalkey111', 'kick-secret-44']) assert.ok(!md.includes(secret), `no ${secret} in the checklist`);
for (const secret of ['a3f9c2leakedkey', 'live_twitchsecret_1', 'yt-secret-2222', 'personalkey111', 'kick-secret-44']) assert.ok(!md.includes(secret), `no ${secret} in the checklist`);
});
});
-t('re-running is idempotent; a held slot is imported once its user has a subject', () => {
- const counts = () => ({
- defs: rt.db.prepare('SELECT COUNT(*) AS n FROM stream_definitions').get().n,
- dests: rt.db.prepare('SELECT COUNT(*) AS n FROM destinations').get().n,
- keys: rt.db.prepare('SELECT COUNT(*) AS n FROM ingest_keys').get().n,
+t('re-running is idempotent; a held slot is imported once its user has a subject', async () => {
+ const counts = async () => ({
+ defs: (await rt.db.prepare('SELECT COUNT(*) AS n FROM stream_definitions').get()).n,
+ dests: (await rt.db.prepare('SELECT COUNT(*) AS n FROM destinations').get()).n,
+ keys: (await rt.db.prepare('SELECT COUNT(*) AS n FROM ingest_keys').get()).n,
});
});
- const before = counts();
+ const before = await counts();
let live = new Database(liveFile, { readonly: true });
let live = new Database(liveFile, { readonly: true });
- const again = migrate({ liveDb: live, rt, apply: true });
+ const again = await migrate({ liveDb: live, rt, apply: true });
live.close();
live.close();
- assert.deepStrictEqual(counts(), before);
+ assert.deepStrictEqual(await counts(), before);
assert.strictEqual(again.counts.imported, 0);
assert.strictEqual(again.counts.imported, 0);
const w = new Database(liveFile);
const w = new Database(liveFile);
- w.prepare("INSERT INTO linked_accounts (user_id, service, service_user_id, subject_id) VALUES (2, 'network', '60', 'usr_01J00000000000000000000D01')").run();
+ await w.prepare("INSERT INTO linked_accounts (user_id, service, service_user_id, subject_id) VALUES (2, 'network', '60', 'usr_01J00000000000000000000D01')").run();
w.close();
w.close();
live = new Database(liveFile, { readonly: true });
live = new Database(liveFile, { readonly: true });
- migrate({ liveDb: live, rt, apply: true });
+ await migrate({ liveDb: live, rt, apply: true });
live.close();
live.close();
- assert.strictEqual(counts().defs, before.defs + 1);
- assert.strictEqual(rt.db.prepare("SELECT status FROM migration_map WHERE source_type = 'managed_stream' AND source_id = '11'").get().status, 'imported');
+ assert.strictEqual((await counts()).defs, before.defs + 1);
+ assert.strictEqual((await rt.db.prepare("SELECT status FROM migration_map WHERE source_type = 'managed_stream' AND source_id = '11'").get()).status, 'imported');
});
});
-t('the CLI runs a dry run against a snapshot and prints the checklist', () => {
+t('the CLI runs a dry run against a snapshot and prints the checklist', async () => {
const d = tmpDir();
const d = tmpDir();
const file = liveSnapshot(d);
const file = liveSnapshot(d);
- const env = testEnv(d);
+ const env = await testEnv(d);
const out = execFileSync(process.execPath, [path.join(ROOT, 'scripts', 'migrate-from-live.js'), '--live-db', file], {
const out = execFileSync(process.execPath, [path.join(ROOT, 'scripts', 'migrate-from-live.js'), '--live-db', file], {
env: { ...process.env, ...env, OPENRE_ENV_FILE: '/nonexistent' }, cwd: ROOT, encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'],
env: { ...process.env, ...env, OPENRE_ENV_FILE: '/nonexistent' }, cwd: ROOT, encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'],
});
});
@@ -1,6 +1,7 @@
@@ -1,6 +1,7 @@
'use strict';
'use strict';
// scripts/cutover-preflight.js against a mocked host: files, commands, DNS, interfaces, the port
// scripts/cutover-preflight.js against a mocked host: files, commands, DNS, interfaces, the port
-// probe and /api/ready are fakes; the three databases are real SQLite files opened read-only.
+// probe and /api/ready are fakes; OpenRe's database is a real PostgreSQL one (PGlite) read in READ ONLY transactions,
+// Events' and Live's are real SQLite files opened read-only (Events' production PostgreSQL has the same tables).
// Covers the host as it is today (2026-09-23: everything the cutover still needs fails), a host
// Covers the host as it is today (2026-09-23: everything the cutover still needs fails), a host
// ready for the first slot, each failure on its own, the off-host port check, the probe URL, the
// ready for the first slot, each failure on its own, the off-host port check, the probe URL, the
// per-slot check, and that no secret value ever reaches the output.
// per-slot check, and that no secret value ever reaches the output.
@@ -9,7 +10,8 @@ const net = require('net');
@@ -9,7 +10,8 @@ const net = require('net');
const path = require('path');
const path = require('path');
const Database = require('better-sqlite3');
const Database = require('better-sqlite3');
const { openDb } = require('../server/db');
const { openDb } = require('../server/db');
-const { preflight, parseArgs, format, parseSsListeners, rtmpProbe, sameSecret, CHECKS } = require('../scripts/cutover-preflight');
+const { load } = require('../server/config');
+const { preflight, parseArgs, format, parseSsListeners, rtmpProbe, sameSecret, withDb, CHECKS } = require('../scripts/cutover-preflight');
const { tmpDir, freePort, suite } = require('./helpers');
const { tmpDir, freePort, suite } = require('./helpers');
const t = suite('preflight');
const t = suite('preflight');
@@ -23,18 +25,18 @@ const LIVE_PID = 4242;
@@ -23,18 +25,18 @@ const LIVE_PID = 4242;
// ── fixtures: three databases ──────────────────────────────────────────────
// ── fixtures: three databases ──────────────────────────────────────────────
-function makeDbs() {
+async function makeDbs() {
const dir = tmpDir();
const dir = tmpDir();
- const openrePath = path.join(dir, 'openre.db');
- const o = openDb(openrePath);
+ // OpenRe's database: PGlite in memory, reached as the target 'pglite:openre' (host() below hands it out).
+ const openrePath = 'pglite:openre';
+ const o = await openDb(load({ NODE_ENV: 'test' }), { memory: true });
const now = Date.now();
const now = Date.now();
const worker = o.prepare(`INSERT INTO workers (id, kind, generation, release, state, endpoints, started_at, heartbeat_at) VALUES (?, ?, ?, ?, ?, '{}', ?, ?)`);
const worker = o.prepare(`INSERT INTO workers (id, kind, generation, release, state, endpoints, started_at, heartbeat_at) VALUES (?, ?, ?, ?, ?, '{}', ?, ?)`);
- worker.run('wrk_1', 'rtmp-ingest', 2, RELEASE, 'ready', now, now);
- worker.run('wrk_2', 'restream', 2, RELEASE, 'ready', now, now);
- o.prepare(`INSERT INTO stream_definitions (id, owner_subject, mirror_to_live, created_at, updated_at) VALUES ('str_1', 'usr_1', 1, ?, ?)`).run(now, now);
- o.prepare(`INSERT INTO external_refs (definition_id, service, type, ref_id, created_at) VALUES ('str_1', 'live', 'managed_stream', '12', ?)`).run(now);
- o.prepare(`INSERT INTO migration_map (source_system, source_type, source_id, target_type, target_id, status, imported_at) VALUES ('live', 'managed_stream', '12', 'stream', 'str_1', 'imported', ?)`).run(now);
- o.close();
+ await worker.run('wrk_1', 'rtmp-ingest', 2, RELEASE, 'ready', now, now);
+ await worker.run('wrk_2', 'restream', 2, RELEASE, 'ready', now, now);
+ await o.prepare(`INSERT INTO stream_definitions (id, owner_subject, mirror_to_live, created_at, updated_at) VALUES ('str_1', 'usr_1', 1, ?, ?)`).run(now, now);
+ await o.prepare(`INSERT INTO external_refs (definition_id, service, type, ref_id, created_at) VALUES ('str_1', 'live', 'managed_stream', '12', ?)`).run(now);
+ await o.prepare(`INSERT INTO migration_map (source_system, source_type, source_id, target_type, target_id, status, imported_at) VALUES ('live', 'managed_stream', '12', 'stream', 'str_1', 'imported', ?)`).run(now);
const eventsPath = path.join(dir, 'events.db');
const eventsPath = path.join(dir, 'events.db');
const e = new Database(eventsPath);
const e = new Database(eventsPath);
@@ -52,18 +54,18 @@ function makeDbs() {
@@ -52,18 +54,18 @@ function makeDbs() {
INSERT INTO managed_streams (id, user_id, slug, title, protocol, streaming_method) VALUES (12, 5, 'test', 'Test', 'rtmp', 'obs');
INSERT INTO managed_streams (id, user_id, slug, title, protocol, streaming_method) VALUES (12, 5, 'test', 'Test', 'rtmp', 'obs');
INSERT INTO linked_accounts (user_id, service, service_user_id, subject_id) VALUES (5, 'network', '77', 'usr_01J0000000000000000000000A');`);
INSERT INTO linked_accounts (user_id, service, service_user_id, subject_id) VALUES (5, 'network', '77', 'usr_01J0000000000000000000000A');`);
l.close();
l.close();
- return { dir, openrePath, eventsPath, livePath };
+ return { dir, openre: o, openrePath, eventsPath, livePath };
}
}
-function addSubscription(dbs, { secret = EVENTS_SECRET, enabled = 1, id = 'sub_1' } = {}) {
+async function addSubscription(dbs, { secret = EVENTS_SECRET, enabled = 1, id = 'sub_1' } = {}) {
const e = new Database(dbs.eventsPath);
const e = new Database(dbs.eventsPath);
- e.prepare(`INSERT INTO subscriptions (id, consumer, topic_pattern, endpoint, secret, enabled, created_at, updated_at) VALUES (?, 'live', 'openre.session.*', 'http://127.0.0.1:3000/internal/openre-events', ?, ?, 0, 0)`).run(id, secret, enabled);
+ await e.prepare(`INSERT INTO subscriptions (id, consumer, topic_pattern, endpoint, secret, enabled, created_at, updated_at) VALUES (?, 'live', 'openre.session.*', 'http://127.0.0.1:3000/internal/openre-events', ?, ?, 0, 0)`).run(id, secret, enabled);
e.close();
e.close();
}
}
-function addDelivery(dbs, status, n = 1) {
+async function addDelivery(dbs, status, n = 1) {
const e = new Database(dbs.eventsPath);
const e = new Database(dbs.eventsPath);
- for (let i = 0; i < n; i++) e.prepare("INSERT INTO deliveries (event_id, subscription_id, seq, priority, status, created_at, updated_at) VALUES (?, 'sub_1', ?, 1, ?, 0, 0)").run(`evt_${status}_${i}`, i, status);
+ for (let i = 0; i < n; i++) await e.prepare("INSERT INTO deliveries (event_id, subscription_id, seq, priority, status, created_at, updated_at) VALUES (?, 'sub_1', ?, 1, ?, 0, 0)").run(`evt_${status}_${i}`, i, status);
e.close();
e.close();
}
}
@@ -72,6 +74,9 @@ function addDelivery(dbs, status, n = 1) {
@@ -72,6 +74,9 @@ function addDelivery(dbs, status, n = 1) {
const envText = (o) => Object.entries(o).filter(([, v]) => v != null).map(([k, v]) => `${k}=${v}`).join('\n') + '\n';
const envText = (o) => Object.entries(o).filter(([, v]) => v != null).map(([k, v]) => `${k}=${v}`).join('\n') + '\n';
/** A host that is ready for the first slot. Every field can be overridden per test. */
/** A host that is ready for the first slot. Every field can be overridden per test. */
+/** The fixture's PGlite handle, which the preflight's close() must not end (every check reads it again). */
+const keepOpen = (db) => new Proxy(db, { get: (t, k) => (k === 'close' ? async () => {} : typeof t[k] === 'function' ? t[k].bind(t) : t[k]) });
+
function host(dbs, over = {}) {
function host(dbs, over = {}) {
const h = {
const h = {
release: RELEASE,
release: RELEASE,
@@ -124,7 +129,7 @@ function host(dbs, over = {}) {
@@ -124,7 +129,7 @@ function host(dbs, over = {}) {
if (h.probeUrl) return { status: h.probeUrl.status || 200, ok: (h.probeUrl.status || 200) < 300, json: async () => h.probeUrl.body };
if (h.probeUrl) return { status: h.probeUrl.status || 200, ok: (h.probeUrl.status || 200) < 300, json: async () => h.probeUrl.body };
throw new Error(`unexpected fetch ${url}`);
throw new Error(`unexpected fetch ${url}`);
},
},
- openDb: h.openDb || ((p) => new Database(p, { readonly: true, fileMustExist: true })),
+ openDb: h.openDb || ((p) => (p === 'pglite:openre' ? keepOpen(dbs.openre) : new Database(p, { readonly: true, fileMustExist: true }))),
};
};
deps.h = h;
deps.h = h;
return deps;
return deps;
@@ -144,8 +149,8 @@ function assertNoSecret(report) {
@@ -144,8 +149,8 @@ function assertNoSecret(report) {
// ── tests ──────────────────────────────────────────────────────────────────
// ── tests ──────────────────────────────────────────────────────────────────
t('a host ready for the first slot: every check passes; on the host the port check is MANUAL', async () => {
t('a host ready for the first slot: every check passes; on the host the port check is MANUAL', async () => {
- const dbs = makeDbs();
- addSubscription(dbs);
+ const dbs = await makeDbs();
+ await addSubscription(dbs);
const deps = host(dbs);
const deps = host(dbs);
const report = await preflight(opts(dbs), deps);
const report = await preflight(opts(dbs), deps);
const r = byId(report);
const r = byId(report);
@@ -164,7 +169,7 @@ t('a host ready for the first slot: every check passes; on the host the port che
@@ -164,7 +169,7 @@ t('a host ready for the first slot: every check passes; on the host the port che
});
});
t('the host as it is on 2026-09-23: old release, loopback bind, Live env and subscription missing', async () => {
t('the host as it is on 2026-09-23: old release, loopback bind, Live env and subscription missing', async () => {
- const dbs = makeDbs();
+ const dbs = await makeDbs();
const deps = host(dbs, {
const deps = host(dbs, {
release: '655b98a10aaa',
release: '655b98a10aaa',
originMain: '450f87c3b2a1',
originMain: '450f87c3b2a1',
@@ -192,7 +197,7 @@ t('the host as it is on 2026-09-23: old release, loopback bind, Live env and sub
@@ -192,7 +197,7 @@ t('the host as it is on 2026-09-23: old release, loopback bind, Live env and sub
});
});
t('release: --expect-release wins over origin/main; --require-commit none skips the ancestry check', async () => {
t('release: --expect-release wins over origin/main; --require-commit none skips the ancestry check', async () => {
- const dbs = makeDbs();
+ const dbs = await makeDbs();
const deps = host(dbs, { originMain: 'bbbbbbbbbbbb', containsCommit: false });
const deps = host(dbs, { originMain: 'bbbbbbbbbbbb', containsCommit: false });
let r = byId(await preflight(opts(dbs, ['--only', 'release', '--expect-release', RELEASE, '--require-commit', 'none']), deps));
let r = byId(await preflight(opts(dbs, ['--only', 'release', '--expect-release', RELEASE, '--require-commit', 'none']), deps));
assert.strictEqual(r.release.status, 'PASS');
assert.strictEqual(r.release.status, 'PASS');
@@ -204,7 +209,7 @@ t('release: --expect-release wins over origin/main; --require-commit none skips
@@ -204,7 +209,7 @@ t('release: --expect-release wins over origin/main; --require-commit none skips
});
});
t('bind: public in the env but the running generation still listens on loopback → deploy.sh workers', async () => {
t('bind: public in the env but the running generation still listens on loopback → deploy.sh workers', async () => {
- const dbs = makeDbs();
+ const dbs = await makeDbs();
const r = byId(await preflight(opts(dbs, ['--only', 'bind']), host(dbs, { ss: 'LISTEN 0 511 127.0.0.1:1936 0.0.0.0:*\n' })));
const r = byId(await preflight(opts(dbs, ['--only', 'bind']), host(dbs, { ss: 'LISTEN 0 511 127.0.0.1:1936 0.0.0.0:*\n' })));
assert.strictEqual(r.bind.status, 'FAIL');
assert.strictEqual(r.bind.status, 'FAIL');
assert.match(r.bind.detail, /deploy\.sh workers/);
assert.match(r.bind.detail, /deploy\.sh workers/);
@@ -213,7 +218,7 @@ t('bind: public in the env but the running generation still listens on loopback
@@ -213,7 +218,7 @@ t('bind: public in the env but the running generation still listens on loopback
});
});
t('env: OPENRE_DRILL in the production env file fails; so do a wrong port or host', async () => {
t('env: OPENRE_DRILL in the production env file fails; so do a wrong port or host', async () => {
- const dbs = makeDbs();
+ const dbs = await makeDbs();
const base = host(dbs).h.openreEnv;
const base = host(dbs).h.openreEnv;
let r = byId(await preflight(opts(dbs, ['--only', 'env']), host(dbs, { openreEnv: { ...base, OPENRE_DRILL: '1' } })));
let r = byId(await preflight(opts(dbs, ['--only', 'env']), host(dbs, { openreEnv: { ...base, OPENRE_DRILL: '1' } })));
assert.strictEqual(r.env.status, 'FAIL');
assert.strictEqual(r.env.status, 'FAIL');
@@ -227,7 +232,7 @@ t('env: OPENRE_DRILL in the production env file fails; so do a wrong port or hos
@@ -227,7 +232,7 @@ t('env: OPENRE_DRILL in the production env file fails; so do a wrong port or hos
});
});
t('dns: a proxied (foreign) address fails; AAAA with an IPv4-only bind warns; off the host --host-ip decides', async () => {
t('dns: a proxied (foreign) address fails; AAAA with an IPv4-only bind warns; off the host --host-ip decides', async () => {
- const dbs = makeDbs();
+ const dbs = await makeDbs();
let r = byId(await preflight(opts(dbs, ['--only', 'dns']), host(dbs, { a4: ['104.21.3.4'] })));
let r = byId(await preflight(opts(dbs, ['--only', 'dns']), host(dbs, { a4: ['104.21.3.4'] })));
assert.strictEqual(r.dns.status, 'FAIL');
assert.strictEqual(r.dns.status, 'FAIL');
assert.match(r.dns.detail, /Cloudflare-proxied/);
assert.match(r.dns.detail, /Cloudflare-proxied/);
@@ -245,7 +250,7 @@ t('dns: a proxied (foreign) address fails; AAAA with an IPv4-only bind warns; of
@@ -245,7 +250,7 @@ t('dns: a proxied (foreign) address fails; AAAA with an IPv4-only bind warns; of
});
});
t('port from outside the host: handshake = PASS, refused = no public listener, timeout = provider edge', async () => {
t('port from outside the host: handshake = PASS, refused = no public listener, timeout = provider edge', async () => {
- const dbs = makeDbs();
+ const dbs = await makeDbs();
const laptop = { wlan0: [{ address: '10.0.0.9', internal: false }] };
const laptop = { wlan0: [{ address: '10.0.0.9', internal: false }] };
const run = (probe) => preflight(opts(dbs, ['--only', 'port']), host(dbs, { interfaces: laptop, probe })).then(byId);
const run = (probe) => preflight(opts(dbs, ['--only', 'port']), host(dbs, { interfaces: laptop, probe })).then(byId);
let r = await run({ ok: true, rtmp: true, ms: 40 });
let r = await run({ ok: true, rtmp: true, ms: 40 });
@@ -262,7 +267,7 @@ t('port from outside the host: handshake = PASS, refused = no public listener, t
@@ -262,7 +267,7 @@ t('port from outside the host: handshake = PASS, refused = no public listener, t
});
});
t('port with --probe-url: open/closed from the probe, anything else MANUAL', async () => {
t('port with --probe-url: open/closed from the probe, anything else MANUAL', async () => {
- const dbs = makeDbs();
+ const dbs = await makeDbs();
const url = 'https://probe.example/check?host={host}&port={port}';
const url = 'https://probe.example/check?host={host}&port={port}';
const run = (probeUrl) => {
const run = (probeUrl) => {
const deps = host(dbs, { probeUrl });
const deps = host(dbs, { probeUrl });
@@ -281,7 +286,7 @@ t('port with --probe-url: open/closed from the probe, anything else MANUAL', asy
@@ -281,7 +286,7 @@ t('port with --probe-url: open/closed from the probe, anything else MANUAL', asy
});
});
t('service: not ready, drill mode, a worker on an old release, no lease, relay off, draining', async () => {
t('service: not ready, drill mode, a worker on an old release, no lease, relay off, draining', async () => {
- const dbs = makeDbs();
+ const dbs = await makeDbs();
const ready = host(dbs).h.ready;
const ready = host(dbs).h.ready;
const run = (body, status = 200, argv = []) => preflight(opts(dbs, ['--only', 'release,service', ...argv]), host(dbs, { ready: { status, body } })).then(byId);
const run = (body, status = 200, argv = []) => preflight(opts(dbs, ['--only', 'release,service', ...argv]), host(dbs, { ready: { status, body } })).then(byId);
let r = await run({ ...ready.body, status: 'not_ready', checks: { db: true, key: false } }, 503);
let r = await run({ ...ready.body, status: 'not_ready', checks: { db: true, key: false } }, 503);
@@ -298,9 +303,7 @@ t('service: not ready, drill mode, a worker on an old release, no lease, relay o
@@ -298,9 +303,7 @@ t('service: not ready, drill mode, a worker on an old release, no lease, relay o
r = await run({ ...ready.body, workers: [{ kind: 'restream', generation: 2, state: 'ready' }] });
r = await run({ ...ready.body, workers: [{ kind: 'restream', generation: 2, state: 'ready' }] });
assert.match(r.service.detail, /0 ready rtmp-ingest workers/);
assert.match(r.service.detail, /0 ready rtmp-ingest workers/);
// the ready generation runs another release than `current`
// the ready generation runs another release than `current`
- const db = new Database(dbs.openrePath);
- db.prepare("UPDATE workers SET release = '655b98a10aaa' WHERE kind = 'rtmp-ingest'").run();
- db.close();
+ await dbs.openre.prepare("UPDATE workers SET release = '655b98a10aaa' WHERE kind = 'rtmp-ingest'").run();
r = await run(ready.body);
r = await run(ready.body);
assert.strictEqual(r.service.status, 'FAIL');
assert.strictEqual(r.service.status, 'FAIL');
assert.match(r.service.detail, /rtmp-ingest#2 runs release 655b98a10aaa, current is aaaaaaaaaaaa: deploy\.sh workers/);
assert.match(r.service.detail, /rtmp-ingest#2 runs release 655b98a10aaa, current is aaaaaaaaaaaa: deploy\.sh workers/);
@@ -309,7 +312,7 @@ t('service: not ready, drill mode, a worker on an old release, no lease, relay o
@@ -309,7 +312,7 @@ t('service: not ready, drill mode, a worker on an old release, no lease, relay o
});
});
t('live-env: Live running without the new settings must be restarted; a short secret is refused', async () => {
t('live-env: Live running without the new settings must be restarted; a short secret is refused', async () => {
- const dbs = makeDbs();
+ const dbs = await makeDbs();
const env = host(dbs).h.liveEnv;
const env = host(dbs).h.liveEnv;
let r = byId(await preflight(opts(dbs, ['--only', 'live-env']), host(dbs, { runningLiveEnv: { OV_OAUTH_CLIENT_SECRET: CLIENT_SECRET } })));
let r = byId(await preflight(opts(dbs, ['--only', 'live-env']), host(dbs, { runningLiveEnv: { OV_OAUTH_CLIENT_SECRET: CLIENT_SECRET } })));
assert.strictEqual(r['live-env'].status, 'FAIL');
assert.strictEqual(r['live-env'].status, 'FAIL');
@@ -324,29 +327,29 @@ t('live-env: Live running without the new settings must be restarted; a short se
@@ -324,29 +327,29 @@ t('live-env: Live running without the new settings must be restarted; a short se
});
});
t('events: wrong secret, disabled, duplicates, dead deliveries fail; waiting deliveries warn; secrets never printed', async () => {
t('events: wrong secret, disabled, duplicates, dead deliveries fail; waiting deliveries warn; secrets never printed', async () => {
- let dbs = makeDbs();
- addSubscription(dbs, { secret: 'x'.repeat(64) });
+ let dbs = await makeDbs();
+ await addSubscription(dbs, { secret: 'x'.repeat(64) });
let r = byId(await preflight(opts(dbs, ['--only', 'events']), host(dbs)));
let r = byId(await preflight(opts(dbs, ['--only', 'events']), host(dbs)));
assert.strictEqual(r.events.status, 'FAIL');
assert.strictEqual(r.events.status, 'FAIL');
assert.match(r.events.detail, /secret is not Live's OPENRE_EVENTS_SECRET/);
assert.match(r.events.detail, /secret is not Live's OPENRE_EVENTS_SECRET/);
assertNoSecret({ results: Object.values(r) });
assertNoSecret({ results: Object.values(r) });
- dbs = makeDbs();
- addSubscription(dbs, { enabled: 0 });
- addDelivery(dbs, 'dead', 2);
+ dbs = await makeDbs();
+ await addSubscription(dbs, { enabled: 0 });
+ await addDelivery(dbs, 'dead', 2);
r = byId(await preflight(opts(dbs, ['--only', 'events']), host(dbs)));
r = byId(await preflight(opts(dbs, ['--only', 'events']), host(dbs)));
assert.match(r.events.detail, /disabled/);
assert.match(r.events.detail, /disabled/);
assert.match(r.events.detail, /2 dead deliveries/);
assert.match(r.events.detail, /2 dead deliveries/);
- dbs = makeDbs();
- addSubscription(dbs);
- addSubscription(dbs, { id: 'sub_2' });
+ dbs = await makeDbs();
+ await addSubscription(dbs);
+ await addSubscription(dbs, { id: 'sub_2' });
r = byId(await preflight(opts(dbs, ['--only', 'events']), host(dbs)));
r = byId(await preflight(opts(dbs, ['--only', 'events']), host(dbs)));
assert.match(r.events.detail, /2 identical subscriptions/);
assert.match(r.events.detail, /2 identical subscriptions/);
- dbs = makeDbs();
- addSubscription(dbs);
- addDelivery(dbs, 'failed', 3);
+ dbs = await makeDbs();
+ await addSubscription(dbs);
+ await addDelivery(dbs, 'failed', 3);
r = byId(await preflight(opts(dbs, ['--only', 'events']), host(dbs)));
r = byId(await preflight(opts(dbs, ['--only', 'events']), host(dbs)));
assert.strictEqual(r.events.status, 'WARN');
assert.strictEqual(r.events.status, 'WARN');
assert.match(r.events.detail, /3 deliveries waiting or retrying/);
assert.match(r.events.detail, /3 deliveries waiting or retrying/);
@@ -356,7 +359,7 @@ t('events: wrong secret, disabled, duplicates, dead deliveries fail; waiting del
@@ -356,7 +359,7 @@ t('events: wrong secret, disabled, duplicates, dead deliveries fail; waiting del
});
});
t('db: integrity_check and foreign keys from a read-only handle', async () => {
t('db: integrity_check and foreign keys from a read-only handle', async () => {
- const dbs = makeDbs();
+ const dbs = await makeDbs();
let r = byId(await preflight(opts(dbs, ['--only', 'db']), host(dbs)));
let r = byId(await preflight(opts(dbs, ['--only', 'db']), host(dbs)));
assert.strictEqual(r.db.status, 'PASS');
assert.strictEqual(r.db.status, 'PASS');
assert.match(r.db.detail, /stream_definitions 1, ingest_keys 0, ingest_sessions 0, destinations 0, migration_map 1/);
assert.match(r.db.detail, /stream_definitions 1, ingest_keys 0, ingest_sessions 0, destinations 0, migration_map 1/);
@@ -367,20 +370,20 @@ t('db: integrity_check and foreign keys from a read-only handle', async () => {
@@ -367,20 +370,20 @@ t('db: integrity_check and foreign keys from a read-only handle', async () => {
r = byId(await preflight(opts(dbs, ['--only', 'db']), host(dbs, { openDb: corrupt })));
r = byId(await preflight(opts(dbs, ['--only', 'db']), host(dbs, { openDb: corrupt })));
assert.strictEqual(r.db.status, 'FAIL');
assert.strictEqual(r.db.status, 'FAIL');
assert.match(r.db.detail, /Page 7 is never used/);
assert.match(r.db.detail, /Page 7 is never used/);
- // the real handle really is read-only
- const ro = new Database(dbs.openrePath, { readonly: true, fileMustExist: true });
- assert.throws(() => ro.prepare("DELETE FROM migration_map").run(), /readonly/);
- ro.close();
+ // the preflight's PostgreSQL transaction really is read-only
+ const w = await withDb(host(dbs), dbs.openrePath, async (db) => await db.prepare('DELETE FROM migration_map').run());
+ assert.match(w.error, /read-only transaction/);
+ assert.strictEqual(Number((await dbs.openre.prepare('SELECT count(*) AS n FROM migration_map').get()).n), 1);
});
});
t('slot: offline RTMP slot with a subject and a migrated definition passes; live now / no subject fail', async () => {
t('slot: offline RTMP slot with a subject and a migrated definition passes; live now / no subject fail', async () => {
- const dbs = makeDbs();
+ const dbs = await makeDbs();
let r = byId(await preflight(opts(dbs, ['--only', 'slot', '--slot', '12']), host(dbs)));
let r = byId(await preflight(opts(dbs, ['--only', 'slot', '--slot', '12']), host(dbs)));
assert.strictEqual(r.slot.status, 'PASS', r.slot.detail);
assert.strictEqual(r.slot.status, 'PASS', r.slot.detail);
assert.match(r.slot.detail, /slot 12 \(@rehearsal, rtmp\/obs\) authority live; OpenRe str_1 \(active, mirrored\); 0 destination\(s\)/);
assert.match(r.slot.detail, /slot 12 \(@rehearsal, rtmp\/obs\) authority live; OpenRe str_1 \(active, mirrored\); 0 destination\(s\)/);
const l = new Database(dbs.livePath);
const l = new Database(dbs.livePath);
- l.prepare('INSERT INTO streams (managed_stream_id, is_live) VALUES (12, 1)').run();
- l.prepare('DELETE FROM linked_accounts').run();
+ await l.prepare('INSERT INTO streams (managed_stream_id, is_live) VALUES (12, 1)').run();
+ await l.prepare('DELETE FROM linked_accounts').run();
l.close();
l.close();
r = byId(await preflight(opts(dbs, ['--only', 'slot', '--slot', '12']), host(dbs)));
r = byId(await preflight(opts(dbs, ['--only', 'slot', '--slot', '12']), host(dbs)));
assert.strictEqual(r.slot.status, 'FAIL');
assert.strictEqual(r.slot.status, 'FAIL');
@@ -25,14 +25,14 @@ async function setup({ media: mediaHandler, recording_mode = 'vod' }) {
@@ -25,14 +25,14 @@ async function setup({ media: mediaHandler, recording_mode = 'vod' }) {
const calls = [];
const calls = [];
const media = await stub((req, res, body) => { calls.push({ method: req.method, url: req.url, body, auth: req.headers.authorization }); mediaHandler(req, res, body, calls); });
const media = await stub((req, res, body) => { calls.push({ method: req.method, url: req.url, body, auth: req.headers.authorization }); mediaHandler(req, res, body, calls); });
const clock = manualClock();
const clock = manualClock();
- const rt = runtime({ clock, env: { MEDIA_URL: media.url, MEDIA_API_KEY: 'k-live', OPENRE_RECORDING_START_DELAY_MS: '0' } });
+ const rt = await runtime({ clock, env: { MEDIA_URL: media.url, MEDIA_API_KEY: 'k-live', OPENRE_RECORDING_START_DELAY_MS: '0' } });
const client = createMediaClient({ config: rt.config });
const client = createMediaClient({ config: rt.config });
const coordinator = createCoordinator({ rt, media: client, log: silent });
const coordinator = createCoordinator({ rt, media: client, log: silent });
- const { definition, key } = rt.store.definitions.create({ owner_subject: OWNER, title: 'Rec', recording_mode, external_refs: [{ service: 'live', type: 'user', id: '8' }] });
- const w = rt.store.workers.register({ kind: 'rtmp-ingest', endpoints: { rtmpPlayPort: 19390, flvPort: 19391 } });
- rt.store.workers.ready(w.id);
- const a = rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker: rt.store.workers.get(w.id) });
- rt.store.sessions.transition(a.session.id, 'live');
+ const { definition, key } = await rt.store.definitions.create({ owner_subject: OWNER, title: 'Rec', recording_mode, external_refs: [{ service: 'live', type: 'user', id: '8' }] });
+ const w = await rt.store.workers.register({ kind: 'rtmp-ingest', endpoints: { rtmpPlayPort: 19390, flvPort: 19391 } });
+ await rt.store.workers.ready(w.id);
+ const a = await rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker: await rt.store.workers.get(w.id) });
+ await rt.store.sessions.transition(a.session.id, 'live');
return { rt, clock, coordinator, session: a.session, calls, media };
return { rt, clock, coordinator, session: a.session, calls, media };
}
}
@@ -52,12 +52,12 @@ t('a live session gets one recording request; finalize when it ends (Media final
@@ -52,12 +52,12 @@ t('a live session gets one recording request; finalize when it ends (Media final
assert.strictEqual(calls[0].body.user_id, 8);
assert.strictEqual(calls[0].body.user_id, 8);
assert.strictEqual(calls[0].body.visibility, 'public');
assert.strictEqual(calls[0].body.visibility, 'public');
assert.strictEqual(calls[1].body.rtmp_url, `rtmp://127.0.0.1:19390/live/${session.id}`);
assert.strictEqual(calls[1].body.rtmp_url, `rtmp://127.0.0.1:19390/live/${session.id}`);
- assert.strictEqual(rt.store.recordings.bySession(session.id).state, 'recording');
- assert.deepStrictEqual(outboxTypes(rt.db).filter(x => x.startsWith('openre.recording')), ['openre.recording.requested']);
- rt.store.sessions.finish(session.id, { reason: 'publisher_disconnected' });
+ assert.strictEqual((await rt.store.recordings.bySession(session.id)).state, 'recording');
+ assert.deepStrictEqual((await outboxTypes(rt.db)).filter(x => x.startsWith('openre.recording')), ['openre.recording.requested']);
+ await rt.store.sessions.finish(session.id, { reason: 'publisher_disconnected' });
await coordinator.tick();
await coordinator.tick();
assert.strictEqual(calls.pop().url, '/api/v1/live/vods/501/finalize');
assert.strictEqual(calls.pop().url, '/api/v1/live/vods/501/finalize');
- assert.strictEqual(rt.store.recordings.bySession(session.id).state, 'finalized');
+ assert.strictEqual((await rt.store.recordings.bySession(session.id)).state, 'finalized');
await media.close();
await media.close();
});
});
@@ -73,15 +73,15 @@ t('Media refusing for disk space: the empty VOD shell is deleted and the request
@@ -73,15 +73,15 @@ t('Media refusing for disk space: the empty VOD shell is deleted and the request
});
});
await coordinator.tick();
await coordinator.tick();
assert.deepStrictEqual(calls.map(c => c.method), ['POST', 'POST', 'DELETE'], 'create, ingest refused, shell deleted');
assert.deepStrictEqual(calls.map(c => c.method), ['POST', 'POST', 'DELETE'], 'create, ingest refused, shell deleted');
- const rec = rt.store.recordings.bySession(session.id);
+ const rec = await rt.store.recordings.bySession(session.id);
assert.strictEqual(rec.state, 'pending');
assert.strictEqual(rec.state, 'pending');
assert.match(rec.last_error, /Disk/);
assert.match(rec.last_error, /Disk/);
assert.ok(rec.next_attempt_at >= clock.now() + 5 * 60 * 1000, 'disk refusals wait five minutes');
assert.ok(rec.next_attempt_at >= clock.now() + 5 * 60 * 1000, 'disk refusals wait five minutes');
refuse = false;
refuse = false;
clock.advance(5 * 60 * 1000 + 1);
clock.advance(5 * 60 * 1000 + 1);
- rt.store.workers.heartbeat(session.worker_id); // the ingest worker is alive throughout
+ await rt.store.workers.heartbeat(session.worker_id); // the ingest worker is alive throughout
await coordinator.tick();
await coordinator.tick();
- assert.strictEqual(rt.store.recordings.bySession(session.id).state, 'recording');
+ assert.strictEqual((await rt.store.recordings.bySession(session.id)).state, 'recording');
await media.close();
await media.close();
});
});
@@ -96,16 +96,16 @@ t('clips-only recordings are deleted after finalising; a session that ends first
@@ -96,16 +96,16 @@ t('clips-only recordings are deleted after finalising; a session that ends first
});
});
await coordinator.tick();
await coordinator.tick();
assert.strictEqual(calls[0].body.clips_only, true);
assert.strictEqual(calls[0].body.clips_only, true);
- rt.store.sessions.finish(session.id, { reason: 'x' });
+ await rt.store.sessions.finish(session.id, { reason: 'x' });
await coordinator.tick();
await coordinator.tick();
assert.deepStrictEqual(calls.slice(-2).map(c => `${c.method} ${c.url}`), ['POST /api/v1/live/vods/700/finalize', 'DELETE /api/v1/live/vods/700']);
assert.deepStrictEqual(calls.slice(-2).map(c => `${c.method} ${c.url}`), ['POST /api/v1/live/vods/700/finalize', 'DELETE /api/v1/live/vods/700']);
await media.close();
await media.close();
const s2 = await setup({ media: (req, res) => reply(res, 500) });
const s2 = await setup({ media: (req, res) => reply(res, 500) });
- s2.rt.store.sessions.finish(s2.session.id, { reason: 'gone' });
- s2.rt.db.prepare("INSERT INTO recordings (id, session_id, mode, state, created_at, updated_at) VALUES ('rec_01J0000000000000000000000A', ?, 'vod', 'pending', 0, 0)").run(s2.session.id);
+ await s2.rt.store.sessions.finish(s2.session.id, { reason: 'gone' });
+ await s2.rt.db.prepare("INSERT INTO recordings (id, session_id, mode, state, created_at, updated_at) VALUES ('rec_01J0000000000000000000000A', ?, 'vod', 'pending', 0, 0)").run(s2.session.id);
await s2.coordinator.tick();
await s2.coordinator.tick();
- assert.strictEqual(s2.rt.store.recordings.bySession(s2.session.id).state, 'cancelled');
+ assert.strictEqual((await s2.rt.store.recordings.bySession(s2.session.id)).state, 'cancelled');
assert.strictEqual(s2.calls.length, 0);
assert.strictEqual(s2.calls.length, 0);
await s2.media.close();
await s2.media.close();
});
});
@@ -127,29 +127,29 @@ t('the relay publishes outbox rows with a Network service token; Events down mea
@@ -127,29 +127,29 @@ t('the relay publishes outbox rows with a Network service token; Events down mea
assert.strictEqual(p.get('audience'), 'openvibe.events');
assert.strictEqual(p.get('audience'), 'openvibe.events');
reply(res, 200, { access_token: 'svc-token', token_type: 'Bearer', expires_in: 300 });
reply(res, 200, { access_token: 'svc-token', token_type: 'Bearer', expires_in: 300 });
});
});
- const rt = runtime({ env: { EVENTS_URL: events.url, OV_NETWORK_INTERNAL_URL: network.url, OV_OAUTH_CLIENT_SECRET: 's3cret' } });
+ const rt = await runtime({ env: { EVENTS_URL: events.url, OV_NETWORK_INTERNAL_URL: network.url, OV_OAUTH_CLIENT_SECRET: 's3cret' } });
assert.strictEqual(rt.events.configured, true);
assert.strictEqual(rt.events.configured, true);
- const { definition } = rt.store.definitions.create({ owner_subject: OWNER });
- rt.store.definitions.rotateKey(definition.id, { rotated_by: OWNER });
+ const { definition } = await rt.store.definitions.create({ owner_subject: OWNER });
+ await rt.store.definitions.rotateKey(definition.id, { rotated_by: OWNER });
await rt.events.outbox.flush();
await rt.events.outbox.flush();
- assert.strictEqual(rt.events.outbox.pending(), 1, 'kept while Events is down');
+ assert.strictEqual(await rt.events.outbox.pending(), 1, 'kept while Events is down');
down = false;
down = false;
- rt.db.prepare('UPDATE event_outbox SET next_attempt_at = 0').run();
+ await rt.db.prepare('UPDATE event_outbox SET next_attempt_at = 0').run();
await rt.events.outbox.flush();
await rt.events.outbox.flush();
- assert.strictEqual(rt.events.outbox.pending(), 0);
+ assert.strictEqual(await rt.events.outbox.pending(), 0);
assert.strictEqual(got[0].event_type, 'openre.key.rotated');
assert.strictEqual(got[0].event_type, 'openre.key.rotated');
assert.strictEqual(got[0].source, 'openre');
assert.strictEqual(got[0].source, 'openre');
await events.close();
await events.close();
await network.close();
await network.close();
});
});
-t('without EVENTS_URL the relay is off but every row is still written with its change', () => {
- const rt = runtime();
+t('without EVENTS_URL the relay is off but every row is still written with its change', async () => {
+ const rt = await runtime();
assert.strictEqual(rt.events.configured, false);
assert.strictEqual(rt.events.configured, false);
assert.strictEqual(rt.events.start(), false);
assert.strictEqual(rt.events.start(), false);
- const { definition } = rt.store.definitions.create({ owner_subject: OWNER });
- rt.store.definitions.rotateKey(definition.id);
- assert.strictEqual(rt.events.status().pending, 1);
+ const { definition } = await rt.store.definitions.create({ owner_subject: OWNER });
+ await rt.store.definitions.rotateKey(definition.id);
+ assert.strictEqual((await rt.events.status()).pending, 1);
});
});
t.run();
t.run();
@@ -13,7 +13,7 @@ const assert = require('assert');
@@ -13,7 +13,7 @@ const assert = require('assert');
const http = require('http');
const http = require('http');
const { spawn } = require('child_process');
const { spawn } = require('child_process');
const {
const {
- tmpDir, testEnv, child, waitFor, request, userToken, freePort, sleep, suite, hasFfmpeg, publish, rtmpSink, OWNER,
+ tmpDir, testEnv, multiProcess, child, waitFor, request, userToken, freePort, sleep, suite, hasFfmpeg, publish, rtmpSink, OWNER,
} = require('./helpers');
} = require('./helpers');
if (!hasFfmpeg()) {
if (!hasFfmpeg()) {
@@ -21,6 +21,10 @@ if (!hasFfmpeg()) {
@@ -21,6 +21,10 @@ if (!hasFfmpeg()) {
process.exit(0);
process.exit(0);
}
}
+if (!multiProcess()) {
+ console.log('rtmp-e2e: skipped (the four OpenRe processes share one database: PostgreSQL only, npm run test:pg)');
+ process.exit(0);
+}
const t = suite('rtmp-e2e');
const t = suite('rtmp-e2e');
const token = userToken({ subjectId: OWNER });
const token = userToken({ subjectId: OWNER });
const procs = [];
const procs = [];
@@ -104,7 +108,7 @@ t('setup: stubs and the four OpenRe processes', async () => {
@@ -104,7 +108,7 @@ t('setup: stubs and the four OpenRe processes', async () => {
});
});
rtmpPort = await freePort();
rtmpPort = await freePort();
const dir = tmpDir();
const dir = tmpDir();
- env = testEnv(dir, {
+ env = await testEnv(dir, {
PORT: String(await freePort()),
PORT: String(await freePort()),
OPENRE_RTMP_PORT: String(rtmpPort),
OPENRE_RTMP_PORT: String(rtmpPort),
OPENRE_RTMP_INTERNAL_PORT_MIN: String(20000 + Math.floor(Math.random() * 20000)),
OPENRE_RTMP_INTERNAL_PORT_MIN: String(20000 + Math.floor(Math.random() * 20000)),
@@ -153,7 +157,7 @@ t('a wrong key is refused; the right key goes live on generation 1', async () =>
@@ -153,7 +157,7 @@ t('a wrong key is refused; the right key goes live on generation 1', async () =>
assert.notStrictEqual(w.code, 0, 'ffmpeg with a wrong key fails');
assert.notStrictEqual(w.code, 0, 'ffmpeg with a wrong key fails');
pubA = publish(`rtmp://127.0.0.1:${rtmpPort}/live/${keyA}`);
pubA = publish(`rtmp://127.0.0.1:${rtmpPort}/live/${keyA}`);
procs.push(pubA);
procs.push(pubA);
- sessionA = await waitFor(() => session(streamA), { what: 'session A live', timeoutMs: 15000 });
+ sessionA = await waitFor(async () => await session(streamA), { what: 'session A live', timeoutMs: 15000 });
assert.strictEqual(sessionA.worker.generation, 1);
assert.strictEqual(sessionA.worker.generation, 1);
});
});
@@ -171,7 +175,7 @@ t('the working destination goes live, the dead one fails, the session stays live
@@ -171,7 +175,7 @@ t('the working destination goes live, the dead one fails, the session stays live
const o = r.body.outputs || [];
const o = r.body.outputs || [];
return o.length === 2 && o.some(x => x.state === 'live') && o.some(x => x.state === 'failed') ? o : null;
return o.length === 2 && o.some(x => x.state === 'live') && o.some(x => x.state === 'failed') ? o : null;
}, { what: 'outputs live + failed', timeoutMs: 40000 });
}, { what: 'outputs live + failed', timeoutMs: 40000 });
- const failed = outputs.find(o => o.state === 'failed');
+ const failed = await outputs.find(o => o.state === 'failed');
assert.match(failed.last_error, /connect|closed|refused|rapid/i);
assert.match(failed.last_error, /connect|closed|refused|rapid/i);
assert.strictEqual((await session(streamA)).id, sessionA.id, 'the source session is still live');
assert.strictEqual((await session(streamA)).id, sessionA.id, 'the source session is still live');
assert.strictEqual(pubA.exitCode, null, 'the encoder is still connected');
assert.strictEqual(pubA.exitCode, null, 'the encoder is still connected');
@@ -201,7 +205,7 @@ t('restarting the API during the broadcast ends nothing; playback works through
@@ -201,7 +205,7 @@ t('restarting the API during the broadcast ends nothing; playback works through
assert.strictEqual(s && s.id, sessionA.id);
assert.strictEqual(s && s.id, sessionA.id);
assert.strictEqual(pubA.exitCode, null);
assert.strictEqual(pubA.exitCode, null);
const outputs = (await request(base, 'GET', `/api/v1/sessions/${sessionA.id}/outputs`, { token })).body.outputs;
const outputs = (await request(base, 'GET', `/api/v1/sessions/${sessionA.id}/outputs`, { token })).body.outputs;
- assert.ok(outputs.some(o => o.state === 'live'), 'the restream kept running');
+ assert.ok(await outputs.some(o => o.state === 'live'), 'the restream kept running');
const head = await new Promise((resolve, reject) => {
const head = await new Promise((resolve, reject) => {
http.get(`${base}/play/${sessionA.id}.flv`, (res) => {
http.get(`${base}/play/${sessionA.id}.flv`, (res) => {
assert.strictEqual(res.statusCode, 200);
assert.strictEqual(res.statusCode, 200);
@@ -224,7 +228,7 @@ t('a new rtmp-ingest generation takes new sessions; generation 1 drains but keep
@@ -224,7 +228,7 @@ t('a new rtmp-ingest generation takes new sessions; generation 1 drains but keep
streamB = r.body.stream.id;
streamB = r.body.stream.id;
pubB = publish(`rtmp://127.0.0.1:${rtmpPort}/live/${r.body.key.key}`);
pubB = publish(`rtmp://127.0.0.1:${rtmpPort}/live/${r.body.key.key}`);
procs.push(pubB);
procs.push(pubB);
- const sB = await waitFor(() => session(streamB), { what: 'session B live', timeoutMs: 15000 });
+ const sB = await waitFor(async () => await session(streamB), { what: 'session B live', timeoutMs: 15000 });
assert.strictEqual(sB.worker.generation, 2, 'new sessions reach the newest generation');
assert.strictEqual(sB.worker.generation, 2, 'new sessions reach the newest generation');
const sA = await session(streamA);
const sA = await session(streamA);
assert.strictEqual(sA.id, sessionA.id);
assert.strictEqual(sA.id, sessionA.id);
@@ -8,12 +8,12 @@ const { NEXT, canTransition } = require('../server/state-machine');
@@ -8,12 +8,12 @@ const { NEXT, canTransition } = require('../server/state-machine');
const t = suite('sessions');
const t = suite('sessions');
-function setup(clock = manualClock()) {
- const rt = runtime({ clock });
- const { definition, key } = rt.store.definitions.create({ owner_subject: OWNER, title: 'Show', external_refs: [{ service: 'live', type: 'managed_stream', id: '12' }] });
- const w = rt.store.workers.register({ kind: 'rtmp-ingest', endpoints: { publicPort: 1936, rtmpPlayPort: 19360, flvPort: 19361 } });
- rt.store.workers.ready(w.id);
- return { rt, clock, definition, key, worker: rt.store.workers.get(w.id) };
+async function setup(clock = manualClock()) {
+ const rt = await runtime({ clock });
+ const { definition, key } = await rt.store.definitions.create({ owner_subject: OWNER, title: 'Show', external_refs: [{ service: 'live', type: 'managed_stream', id: '12' }] });
+ const w = await rt.store.workers.register({ kind: 'rtmp-ingest', endpoints: { publicPort: 1936, rtmpPlayPort: 19360, flvPort: 19361 } });
+ await rt.store.workers.ready(w.id);
+ return { rt, clock, definition, key, worker: await rt.store.workers.get(w.id) };
}
}
t('the transition table is exactly starting→live→ending→ended with failed from any open state', () => {
t('the transition table is exactly starting→live→ending→ended with failed from any open state', () => {
@@ -23,18 +23,18 @@ t('the transition table is exactly starting→live→ending→ended with failed
@@ -23,18 +23,18 @@ t('the transition table is exactly starting→live→ending→ended with failed
assert.ok(!canTransition('failed', 'ended'));
assert.ok(!canTransition('failed', 'ended'));
});
});
-t('a clean session emits started then ended; envelopes are valid events.event-envelope@1', () => {
- const { rt, clock, definition, key, worker } = setup();
- const a = rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker });
+t('a clean session emits started then ended; envelopes are valid events.event-envelope@1', async () => {
+ const { rt, clock, definition, key, worker } = await setup();
+ const a = await rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker });
assert.strictEqual(a.session.state, 'starting');
assert.strictEqual(a.session.state, 'starting');
- assert.deepStrictEqual(outboxTypes(rt.db), [], 'starting is internal');
- assert.ok(rt.store.sessions.transition(a.session.id, 'live', { reason: 'media_flowing' }).ok);
+ assert.deepStrictEqual(await outboxTypes(rt.db), [], 'starting is internal');
+ assert.ok((await rt.store.sessions.transition(a.session.id, 'live', { reason: 'media_flowing' })).ok);
clock.advance(90 * 1000);
clock.advance(90 * 1000);
- const f = rt.store.sessions.finish(a.session.id, { reason: 'publisher_disconnected' });
+ const f = await rt.store.sessions.finish(a.session.id, { reason: 'publisher_disconnected' });
assert.ok(f.ok);
assert.ok(f.ok);
assert.strictEqual(f.session.state, 'ended');
assert.strictEqual(f.session.state, 'ended');
- assert.deepStrictEqual(outboxTypes(rt.db), ['openre.session.started', 'openre.session.ended']);
- const [started, ended] = outboxEnvelopes(rt.db);
+ assert.deepStrictEqual(await outboxTypes(rt.db), ['openre.session.started', 'openre.session.ended']);
+ const [started, ended] = await outboxEnvelopes(rt.db);
for (const ev of [started, ended]) {
for (const ev of [started, ended]) {
const v = contracts.validate('events.event-envelope@1', ev);
const v = contracts.validate('events.event-envelope@1', ev);
assert.ok(v.valid, JSON.stringify(v.errors));
assert.ok(v.valid, JSON.stringify(v.errors));
@@ -47,78 +47,78 @@ t('a clean session emits started then ended; envelopes are valid events.event-en
@@ -47,78 +47,78 @@ t('a clean session emits started then ended; envelopes are valid events.event-en
assert.deepStrictEqual(started.payload.external_refs, [{ service: 'live', type: 'managed_stream', id: '12', label: null }]);
assert.deepStrictEqual(started.payload.external_refs, [{ service: 'live', type: 'managed_stream', id: '12', label: null }]);
assert.strictEqual(started.payload.playback.flv.internal_url, `http://127.0.0.1:19361/live/${a.session.id}.flv`);
assert.strictEqual(started.payload.playback.flv.internal_url, `http://127.0.0.1:19361/live/${a.session.id}.flv`);
assert.strictEqual(ended.payload.duration_seconds, 90);
assert.strictEqual(ended.payload.duration_seconds, 90);
- assert.deepStrictEqual(rt.store.sessions.transitions(a.session.id).map(x => x.to_state), ['starting', 'live', 'ending', 'ended']);
+ assert.deepStrictEqual((await rt.store.sessions.transitions(a.session.id)).map(x => x.to_state), ['starting', 'live', 'ending', 'ended']);
});
});
-t('a publisher that leaves during the handshake ends without events; failures always emit', () => {
- const { rt, definition, key, worker } = setup();
- const a = rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker });
- rt.store.sessions.finish(a.session.id, { reason: 'publisher_disconnected' });
- assert.deepStrictEqual(outboxTypes(rt.db), []);
- const b = rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker });
- rt.store.sessions.transition(b.session.id, 'failed', { reason: 'worker_lost' });
- const ev = outboxEnvelopes(rt.db).pop();
+t('a publisher that leaves during the handshake ends without events; failures always emit', async () => {
+ const { rt, definition, key, worker } = await setup();
+ const a = await rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker });
+ await rt.store.sessions.finish(a.session.id, { reason: 'publisher_disconnected' });
+ assert.deepStrictEqual(await outboxTypes(rt.db), []);
+ const b = await rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker });
+ await rt.store.sessions.transition(b.session.id, 'failed', { reason: 'worker_lost' });
+ const ev = (await outboxEnvelopes(rt.db)).pop();
assert.strictEqual(ev.event_type, 'openre.session.failed');
assert.strictEqual(ev.event_type, 'openre.session.failed');
assert.strictEqual(ev.payload.was_live, false);
assert.strictEqual(ev.payload.was_live, false);
assert.strictEqual(ev.payload.failure_reason, 'worker_lost');
assert.strictEqual(ev.payload.failure_reason, 'worker_lost');
});
});
-t('invalid transitions change nothing (a worker and the coordinator racing)', () => {
- const { rt, definition, key, worker } = setup();
- const a = rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker });
- rt.store.sessions.transition(a.session.id, 'live');
- rt.store.sessions.transition(a.session.id, 'failed', { reason: 'lease_expired' });
- const before = outboxTypes(rt.db).length;
- const r = rt.store.sessions.finish(a.session.id, { reason: 'publisher_disconnected' });
+t('invalid transitions change nothing (a worker and the coordinator racing)', async () => {
+ const { rt, definition, key, worker } = await setup();
+ const a = await rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker });
+ await rt.store.sessions.transition(a.session.id, 'live');
+ await rt.store.sessions.transition(a.session.id, 'failed', { reason: 'lease_expired' });
+ const before = (await outboxTypes(rt.db)).length;
+ const r = await rt.store.sessions.finish(a.session.id, { reason: 'publisher_disconnected' });
assert.strictEqual(r.ok, false);
assert.strictEqual(r.ok, false);
assert.strictEqual(r.code, 'invalid_transition');
assert.strictEqual(r.code, 'invalid_transition');
- assert.strictEqual(rt.store.sessions.get(a.session.id).state, 'failed');
- assert.strictEqual(outboxTypes(rt.db).length, before);
+ assert.strictEqual((await rt.store.sessions.get(a.session.id)).state, 'failed');
+ assert.strictEqual((await outboxTypes(rt.db)).length, before);
});
});
-t('one open session per stream: a second publisher is refused while the first holds its lease', () => {
- const { rt, clock, definition, key, worker } = setup();
- const a = rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker });
- assert.strictEqual(rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker }).error, 'duplicate_publisher');
+t('one open session per stream: a second publisher is refused while the first holds its lease', async () => {
+ const { rt, clock, definition, key, worker } = await setup();
+ const a = await rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker });
+ assert.strictEqual((await rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker })).error, 'duplicate_publisher');
// Lease expired (worker died): the stale session fails and the new publish is admitted.
// Lease expired (worker died): the stale session fails and the new publish is admitted.
clock.advance(rt.config.workers.leaseMs + 1);
clock.advance(rt.config.workers.leaseMs + 1);
- const b = rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker });
+ const b = await rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker });
assert.ok(b.session);
assert.ok(b.session);
- assert.strictEqual(rt.store.sessions.get(a.session.id).state, 'failed');
- assert.strictEqual(rt.store.sessions.get(a.session.id).failure_reason, 'lease_expired');
+ assert.strictEqual((await rt.store.sessions.get(a.session.id)).state, 'failed');
+ assert.strictEqual((await rt.store.sessions.get(a.session.id)).failure_reason, 'lease_expired');
});
});
-t('heartbeats renew the lease of every session the worker owns', () => {
- const { rt, clock, definition, key, worker } = setup();
- const a = rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker });
- const lease1 = rt.store.sessions.get(a.session.id).lease_expires_at;
+t('heartbeats renew the lease of every session the worker owns', async () => {
+ const { rt, clock, definition, key, worker } = await setup();
+ const a = await rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker });
+ const lease1 = (await rt.store.sessions.get(a.session.id)).lease_expires_at;
clock.advance(5000);
clock.advance(5000);
- rt.store.workers.heartbeat(worker.id);
- assert.strictEqual(rt.store.sessions.get(a.session.id).lease_expires_at, lease1 + 5000);
+ await rt.store.workers.heartbeat(worker.id);
+ assert.strictEqual((await rt.store.sessions.get(a.session.id)).lease_expires_at, lease1 + 5000);
});
});
-t('ending a session asks its outputs to stop; an output never moves the session', () => {
- const { rt, definition, key, worker } = setup();
- const dest = rt.store.outputs.createDestination(definition.id, { platform: 'custom', server_url: 'rtmp://127.0.0.1:1/app', stream_key: 'k1234' });
- rt.store.outputs.createDestination(definition.id, { platform: 'custom', server_url: 'rtmp://127.0.0.1:2/app', stream_key: 'k5678' });
- const a = rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker });
- rt.store.sessions.transition(a.session.id, 'live');
- assert.strictEqual(rt.store.outputs.ensureAutoOutputs(), 2);
- const [out, other] = rt.store.outputs.outputsOfSession(a.session.id);
- rt.store.outputs.report(other.id, { state: 'live' });
- rt.store.outputs.report(out.id, { state: 'failed', last_error: 'boom' });
- assert.strictEqual(rt.store.sessions.get(a.session.id).state, 'live', 'a failed output leaves the session live');
- assert.ok(outboxTypes(rt.db).includes('openre.output.failed'));
- rt.store.sessions.finish(a.session.id, { reason: 'publisher_disconnected' });
- assert.strictEqual(rt.db.prepare('SELECT desired FROM outputs WHERE id = ?').get(other.id).desired, 'stop', 'the running output is told to stop');
+t('ending a session asks its outputs to stop; an output never moves the session', async () => {
+ const { rt, definition, key, worker } = await setup();
+ const dest = await rt.store.outputs.createDestination(definition.id, { platform: 'custom', server_url: 'rtmp://127.0.0.1:1/app', stream_key: 'k1234' });
+ await rt.store.outputs.createDestination(definition.id, { platform: 'custom', server_url: 'rtmp://127.0.0.1:2/app', stream_key: 'k5678' });
+ const a = await rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker });
+ await rt.store.sessions.transition(a.session.id, 'live');
+ assert.strictEqual(await rt.store.outputs.ensureAutoOutputs(), 2);
+ const [out, other] = await rt.store.outputs.outputsOfSession(a.session.id);
+ await rt.store.outputs.report(other.id, { state: 'live' });
+ await rt.store.outputs.report(out.id, { state: 'failed', last_error: 'boom' });
+ assert.strictEqual((await rt.store.sessions.get(a.session.id)).state, 'live', 'a failed output leaves the session live');
+ assert.ok((await outboxTypes(rt.db)).includes('openre.output.failed'));
+ await rt.store.sessions.finish(a.session.id, { reason: 'publisher_disconnected' });
+ assert.strictEqual((await rt.db.prepare('SELECT desired FROM outputs WHERE id = ?').get(other.id)).desired, 'stop', 'the running output is told to stop');
assert.ok(dest.id);
assert.ok(dest.id);
});
});
-t('an end request is recorded for the owning worker to act on', () => {
- const { rt, definition, key, worker } = setup();
- const a = rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker });
- assert.ok(rt.store.sessions.requestEnd(a.session.id, OWNER));
- assert.deepStrictEqual(rt.store.sessions.endRequestsFor(worker.id).map(s => s.id), [a.session.id]);
+t('an end request is recorded for the owning worker to act on', async () => {
+ const { rt, definition, key, worker } = await setup();
+ const a = await rt.store.sessions.admit({ definition, key, protocol: 'rtmp', worker });
+ assert.ok(await rt.store.sessions.requestEnd(a.session.id, OWNER));
+ assert.deepStrictEqual((await rt.store.sessions.endRequestsFor(worker.id)).map(s => s.id), [a.session.id]);
});
});
t.run();
t.run();
@@ -40,7 +40,7 @@ t('setup: a definition made for Live slot 12, as Live does at the switch', async
@@ -40,7 +40,7 @@ t('setup: a definition made for Live slot 12, as Live does at the switch', async
assert.strictEqual(created.status, 201);
assert.strictEqual(created.status, 201);
streamId = created.body.stream.id;
streamId = created.body.stream.id;
key = created.body.key.key;
key = created.body.key.key;
- assert.ok(api.rt.store.definitions.resolveIngestKey(key, 'rtmp').definition);
+ assert.ok((await api.rt.store.definitions.resolveIngestKey(key, 'rtmp')).definition);
});
});
t('--state disabled: the definition is disabled and its key refused at the handshake', async () => {
t('--state disabled: the definition is disabled and its key refused at the handshake', async () => {
@@ -48,7 +48,7 @@ t('--state disabled: the definition is disabled and its key refused at the hands
@@ -48,7 +48,7 @@ t('--state disabled: the definition is disabled and its key refused at the hands
const r = await run({ slot: '12', state: 'disabled', env, fetchImpl, log });
const r = await run({ slot: '12', state: 'disabled', env, fetchImpl, log });
assert.deepStrictEqual(r, { id: streamId, state: 'disabled', changed: true, open_session: null });
assert.deepStrictEqual(r, { id: streamId, state: 'disabled', changed: true, open_session: null });
assert.deepStrictEqual(tokenRequests.at(-1), { grant_type: 'client_credentials', client_id: 'live', client_secret: CLIENT_SECRET, audience: 'openvibe.openre' });
assert.deepStrictEqual(tokenRequests.at(-1), { grant_type: 'client_credentials', client_id: 'live', client_secret: CLIENT_SECRET, audience: 'openvibe.openre' });
- assert.deepStrictEqual(api.rt.store.definitions.resolveIngestKey(key, 'rtmp'), { error: 'stream_disabled' });
+ assert.deepStrictEqual(await api.rt.store.definitions.resolveIngestKey(key, 'rtmp'), { error: 'stream_disabled' });
assert.match(log.lines.join('\n'), new RegExp(`${streamId} \\(live:managed_stream:12\\): active → disabled`));
assert.match(log.lines.join('\n'), new RegExp(`${streamId} \\(live:managed_stream:12\\): active → disabled`));
for (const l of log.lines) { assert.ok(!l.includes(key)); assert.ok(!l.includes(CLIENT_SECRET)); }
for (const l of log.lines) { assert.ok(!l.includes(key)); assert.ok(!l.includes(CLIENT_SECRET)); }
const again = await run({ slot: '12', state: 'disabled', env, fetchImpl, log });
const again = await run({ slot: '12', state: 'disabled', env, fetchImpl, log });
@@ -58,7 +58,7 @@ t('--state disabled: the definition is disabled and its key refused at the hands
@@ -58,7 +58,7 @@ t('--state disabled: the definition is disabled and its key refused at the hands
t('--state active: the same key works again (re-switching the slot later)', async () => {
t('--state active: the same key works again (re-switching the slot later)', async () => {
const r = await run({ slot: '12', state: 'active', env, fetchImpl, log: () => {} });
const r = await run({ slot: '12', state: 'active', env, fetchImpl, log: () => {} });
assert.strictEqual(r.state, 'active');
assert.strictEqual(r.state, 'active');
- assert.ok(api.rt.store.definitions.resolveIngestKey(key, 'rtmp').definition);
+ assert.ok((await api.rt.store.definitions.resolveIngestKey(key, 'rtmp')).definition);
});
});
t('a slot without a definition is an error; arguments are checked', async () => {
t('a slot without a definition is an error; arguments are checked', async () => {
@@ -9,26 +9,28 @@
@@ -9,26 +9,28 @@
const path = require('path');
const path = require('path');
if (require.main === module) {
if (require.main === module) {
- require('dotenv').config({ path: process.env.OPENRE_ENV_FILE || path.join(process.cwd(), '.env') });
- const { load, exitIfDrill } = require('../server/config');
- const { openRuntime } = require('../server/store');
- const { createCoordinator } = require('../server/coordinator');
- const { createMediaClient } = require('../server/media-client');
- const config = load();
- exitIfDrill(config, 'openre-session-coordinator');
- const rt = openRuntime({ config });
- const media = createMediaClient({ config });
- const lineage = require('../server/lineage').createLineage({ db: rt.db, config });
- const coordinator = createCoordinator({ rt, media, lineage });
- coordinator.start();
- console.log(`[coordinator] running (events relay ${rt.events.configured ? 'on' : 'off: EVENTS_URL/OV_OAUTH_CLIENT_SECRET not set'}, recordings ${media.configured ? `on → ${config.media.url} app ${config.media.appId}` : 'off'}, Live lineage ${lineage.enabled ? 'on' : 'off'})`);
- const prune = setInterval(() => { try { rt.events.outbox.prune(); } catch { /* next time */ } }, 6 * 60 * 60 * 1000);
- prune.unref();
- const shutdown = (sig) => {
- console.log(`[coordinator] ${sig}: stopping`);
- coordinator.stop().then(() => { rt.db.close(); process.exit(0); }, () => process.exit(1));
- setTimeout(() => process.exit(1), 10000).unref();
- };
- process.on('SIGTERM', () => shutdown('SIGTERM'));
- process.on('SIGINT', () => shutdown('SIGINT'));
+ (async () => {
+ require('dotenv').config({ path: process.env.OPENRE_ENV_FILE || path.join(process.cwd(), '.env') });
+ const { load, exitIfDrill } = require('../server/config');
+ const { openRuntime } = require('../server/store');
+ const { createCoordinator } = require('../server/coordinator');
+ const { createMediaClient } = require('../server/media-client');
+ const config = load();
+ exitIfDrill(config, 'openre-session-coordinator');
+ const rt = await openRuntime({ config });
+ const media = createMediaClient({ config });
+ const lineage = require('../server/lineage').createLineage({ db: rt.db, config });
+ const coordinator = createCoordinator({ rt, media, lineage });
+ coordinator.start();
+ console.log(`[coordinator] running (events relay ${rt.events.configured ? 'on' : 'off: EVENTS_URL/OV_OAUTH_CLIENT_SECRET not set'}, recordings ${media.configured ? `on → ${config.media.url} app ${config.media.appId}` : 'off'}, Live lineage ${lineage.enabled ? 'on' : 'off'})`);
+ const prune = setInterval(() => { try { rt.events.outbox.prune(); } catch { /* next time */ } }, 6 * 60 * 60 * 1000);
+ prune.unref();
+ const shutdown = (sig) => {
+ console.log(`[coordinator] ${sig}: stopping`);
+ coordinator.stop().then(() => { rt.db.close(); process.exit(0); }, () => process.exit(1));
+ setTimeout(() => process.exit(1), 10000).unref();
+ };
+ process.on('SIGTERM', () => shutdown('SIGTERM'));
+ process.on('SIGINT', () => shutdown('SIGINT'));
+ })().catch((err) => { console.error(`[coordinator] failed to start: ${err && err.stack || err}`); process.exit(1); });
}
}
@@ -1,3 +1,3 @@
@@ -1,3 +1,3 @@
'use strict';
'use strict';
/** openre-jsmpeg: worker interface only, the transport is not ported yet (see unported-transport.js). */
/** openre-jsmpeg: worker interface only, the transport is not ported yet (see unported-transport.js). */
-if (require.main === module) require('./unported-transport').main('jsmpeg');
+if (require.main === module) require('./unported-transport').main('jsmpeg').catch((err) => { console.error(`[jsmpeg] failed to start: ${err && err.stack || err}`); process.exit(1); });
@@ -24,39 +24,39 @@ function createRestreamWorker({ rt, log = console, exit = (code) => process.exit
@@ -24,39 +24,39 @@ function createRestreamWorker({ rt, log = console, exit = (code) => process.exit
const runtime = createWorkerRuntime({
const runtime = createWorkerRuntime({
rt, kind: 'restream', log, exit,
rt, kind: 'restream', log, exit,
hooks: {
hooks: {
- onDrainDeadline: () => handOver(),
+ onDrainDeadline: async () => await handOver(),
onLost: () => stopAll('worker lost'),
onLost: () => stopAll('worker lost'),
activeCount: () => runners.size,
activeCount: () => runners.size,
- onExit: () => { closing = true; clearTimeout(pollTimer); if (runtime.me) store.outputs.release(runtime.me.id); },
+ onExit: async () => { closing = true; clearTimeout(pollTimer); if (runtime.me) await store.outputs.release(runtime.me.id); },
},
},
});
});
- function inputUrlFor(session) {
+ async function inputUrlFor(session) {
if (session.protocol !== 'rtmp') return null;
if (session.protocol !== 'rtmp') return null;
- const pb = store.sessions.playback(session);
+ const pb = await store.sessions.playback(session);
return pb && pb.flv ? pb.flv.internal_url : null;
return pb && pb.flv ? pb.flv.internal_url : null;
}
}
- function poll() {
+ async function poll() {
if (closing || !runtime.me) return;
if (closing || !runtime.me) return;
- const rows = store.outputs.forWorker(runtime.me.id);
+ const rows = await store.outputs.forWorker(runtime.me.id);
const seen = new Set();
const seen = new Set();
for (const row of rows) {
for (const row of rows) {
seen.add(row.id);
seen.add(row.id);
let runner = runners.get(row.id);
let runner = runners.get(row.id);
- const session = store.sessions.get(row.session_id);
+ const session = await store.sessions.get(row.session_id);
const shouldRun = row.desired === 'run' && session && session.state === 'live';
const shouldRun = row.desired === 'run' && session && session.state === 'live';
if (!shouldRun) {
if (!shouldRun) {
- if (runner) { runner.stop(row.desired === 'stop' ? 'stop requested' : 'session not live'); runners.delete(row.id); } else store.outputs.report(row.id, { state: 'stopped', ended_at: Date.now() }, { workerId: runtime.me.id });
+ if (runner) { runner.stop(row.desired === 'stop' ? 'stop requested' : 'session not live'); runners.delete(row.id); } else await store.outputs.report(row.id, { state: 'stopped', ended_at: Date.now() }, { workerId: runtime.me.id });
continue;
continue;
}
}
if (runner && runner.stopped) { runners.delete(row.id); runner = null; continue; }
if (runner && runner.stopped) { runners.delete(row.id); runner = null; continue; }
if (runner) continue;
if (runner) continue;
// A draining generation starts nothing new: an output assigned to it but not started yet
// A draining generation starts nothing new: an output assigned to it but not started yet
// goes back to the coordinator for the newest generation.
// goes back to the coordinator for the newest generation.
- if (runtime.draining) { store.outputs.unassign(row.id); seen.delete(row.id); continue; }
- const input = inputUrlFor(session);
- if (!input) { store.outputs.report(row.id, { state: 'failed', last_error: `restream from ${session.protocol} sessions is not supported by OpenRe yet`, ended_at: Date.now() }, { workerId: runtime.me.id }); continue; }
+ if (runtime.draining) { await store.outputs.unassign(row.id); seen.delete(row.id); continue; }
+ const input = await inputUrlFor(session);
+ if (!input) { await store.outputs.report(row.id, { state: 'failed', last_error: `restream from ${session.protocol} sessions is not supported by OpenRe yet`, ended_at: Date.now() }, { workerId: runtime.me.id }); continue; }
// Let the ingest settle before pulling (Live waits 3 s for node-media-server's FLV).
// Let the ingest settle before pulling (Live waits 3 s for node-media-server's FLV).
if (session.live_at && Date.now() - session.live_at < config.outputs.startDelayMs) continue;
if (session.live_at && Date.now() - session.live_at < config.outputs.startDelayMs) continue;
runner = new OutputRunner({ outputId: row.id, destinationId: row.destination_id, inputUrl: input, store, config, log, spawnImpl, lookup, workerId: runtime.me.id });
runner = new OutputRunner({ outputId: row.id, destinationId: row.destination_id, inputUrl: input, store, config, log, spawnImpl, lookup, workerId: runtime.me.id });
@@ -73,8 +73,8 @@ function createRestreamWorker({ rt, log = console, exit = (code) => process.exit
@@ -73,8 +73,8 @@ function createRestreamWorker({ rt, log = console, exit = (code) => process.exit
function schedule() {
function schedule() {
clearTimeout(pollTimer);
clearTimeout(pollTimer);
if (closing) return;
if (closing) return;
- pollTimer = setTimeout(() => {
- try { poll(); } catch (err) { log.error(`[restream] poll failed: ${err.stack || err}`); }
+ pollTimer = setTimeout(async () => {
+ try { await poll(); } catch (err) { log.error(`[restream] poll failed: ${err.stack || err}`); }
schedule();
schedule();
}, config.workers.restreamPollMs);
}, config.workers.restreamPollMs);
}
}
@@ -84,15 +84,15 @@ function createRestreamWorker({ rt, log = console, exit = (code) => process.exit
@@ -84,15 +84,15 @@ function createRestreamWorker({ rt, log = console, exit = (code) => process.exit
}
}
/** Drain deadline: stop here and let the coordinator give the outputs to the newest generation. */
/** Drain deadline: stop here and let the coordinator give the outputs to the newest generation. */
- function handOver() {
+ async function handOver() {
for (const [id, r] of runners) { r.stopped = true; r.clearTimers(); if (r.proc) { try { r.proc.kill('SIGTERM'); } catch { /* */ } } runners.delete(id); }
for (const [id, r] of runners) { r.stopped = true; r.clearTimers(); if (r.proc) { try { r.proc.kill('SIGTERM'); } catch { /* */ } } runners.delete(id); }
- const n = store.outputs.release(runtime.me.id);
+ const n = await store.outputs.release(runtime.me.id);
log.log(`[restream] drain deadline: released ${n} output(s) to the newest generation`);
log.log(`[restream] drain deadline: released ${n} output(s) to the newest generation`);
}
}
- function start() {
- runtime.register({});
- runtime.ready();
+ async function start() {
+ await runtime.register({});
+ await runtime.ready();
schedule();
schedule();
log.log(`[restream] generation ${runtime.me.generation} ready`);
log.log(`[restream] generation ${runtime.me.generation} ready`);
return runtime.me;
return runtime.me;
@@ -103,26 +103,28 @@ function createRestreamWorker({ rt, log = console, exit = (code) => process.exit
@@ -103,26 +103,28 @@ function createRestreamWorker({ rt, log = console, exit = (code) => process.exit
poll,
poll,
runtime,
runtime,
runners: () => runners,
runners: () => runners,
- drain: (reason) => runtime.drainNow(reason),
+ drain: async (reason) => await runtime.drainNow(reason),
close() { closing = true; clearTimeout(pollTimer); stopAll('closing'); runtime.stop(); },
close() { closing = true; clearTimeout(pollTimer); stopAll('closing'); runtime.stop(); },
};
};
}
}
if (require.main === module) {
if (require.main === module) {
- require('dotenv').config({ path: process.env.OPENRE_ENV_FILE || path.join(process.cwd(), '.env') });
- const { load, exitIfDrill } = require('../server/config');
- const { openRuntime } = require('../server/store');
- const config = load();
- exitIfDrill(config, 'openre-restream-worker');
- const rt = openRuntime({ config });
- const worker = createRestreamWorker({ rt });
- worker.start();
- // If this process dies, its ffmpeg children must not keep pushing unsupervised: the outputs are
- // reassigned to another worker, and two pushes to one ingest make platforms drop both.
- process.on('exit', () => {
- for (const r of worker.runners().values()) { try { if (r.proc) r.proc.kill('SIGKILL'); } catch { /* */ } }
- });
- for (const sig of ['SIGTERM', 'SIGINT']) process.on(sig, () => worker.drain(sig));
+ (async () => {
+ require('dotenv').config({ path: process.env.OPENRE_ENV_FILE || path.join(process.cwd(), '.env') });
+ const { load, exitIfDrill } = require('../server/config');
+ const { openRuntime } = require('../server/store');
+ const config = load();
+ exitIfDrill(config, 'openre-restream-worker');
+ const rt = await openRuntime({ config });
+ const worker = createRestreamWorker({ rt });
+ worker.start();
+ // If this process dies, its ffmpeg children must not keep pushing unsupervised: the outputs are
+ // reassigned to another worker, and two pushes to one ingest make platforms drop both.
+ process.on('exit', () => {
+ for (const r of worker.runners().values()) { try { if (r.proc) r.proc.kill('SIGKILL'); } catch { /* */ } }
+ });
+ for (const sig of ['SIGTERM', 'SIGINT']) process.on(sig, () => worker.drain(sig));
+ })().catch((err) => { console.error(`[restream-worker] failed to start: ${err && err.stack || err}`); process.exit(1); });
}
}
module.exports = { createRestreamWorker };
module.exports = { createRestreamWorker };
@@ -42,8 +42,8 @@ class OutputRunner {
@@ -42,8 +42,8 @@ class OutputRunner {
return t;
return t;
}
}
- report(change) { try { return this.store.outputs.report(this.outputId, change, { workerId: this.workerId }); } catch (err) { this.log.error(`[restream] report ${this.outputId}: ${err.message}`); return null; } }
- note(level, message) { try { this.store.outputs.log(this.outputId, this.destinationId, level, this.scrub(message)); } catch { /* logs are best effort */ } }
+ async report(change) { try { return await this.store.outputs.report(this.outputId, change, { workerId: this.workerId }); } catch (err) { this.log.error(`[restream] report ${this.outputId}: ${err.message}`); return null; } }
+ async note(level, message) { try { await this.store.outputs.log(this.outputId, this.destinationId, level, this.scrub(message)); } catch { /* logs are best effort */ } }
/** ffmpeg's stderr can echo the output URL: never let a destination key reach a log or event. */
/** ffmpeg's stderr can echo the output URL: never let a destination key reach a log or event. */
scrub(text) {
scrub(text) {
@@ -55,7 +55,7 @@ class OutputRunner {
@@ -55,7 +55,7 @@ class OutputRunner {
async start() {
async start() {
if (this.stopped) return;
if (this.stopped) return;
let dest;
let dest;
- try { dest = this.store.outputs.destinationForWorker(this.destinationId); } catch (err) {
+ try { dest = await this.store.outputs.destinationForWorker(this.destinationId); } catch (err) {
return this.fail(`destination secrets unavailable: ${err.message}`, { cooldown: false });
return this.fail(`destination secrets unavailable: ${err.message}`, { cooldown: false });
}
}
if (!dest || !dest.enabled || dest.hold_reason) return this.stop('destination disabled');
if (!dest || !dest.enabled || dest.hold_reason) return this.stop('destination disabled');
@@ -139,13 +139,13 @@ class OutputRunner {
@@ -139,13 +139,13 @@ class OutputRunner {
}, this.o.liveAckTimeoutMs);
}, this.o.liveAckTimeoutMs);
}
}
- confirmLive(proc) {
+ async confirmLive(proc) {
if (this.proc !== proc || this.liveThisRun || this.stopped) return;
if (this.proc !== proc || this.liveThisRun || this.stopped) return;
this.liveThisRun = true;
this.liveThisRun = true;
this.everLive = true;
this.everLive = true;
this.rapidCrashCount = 0;
this.rapidCrashCount = 0;
this.status = 'live';
this.status = 'live';
- try { this.store.outputs.clearDestinationCooldown(this.destinationId); } catch { /* */ }
+ try { await this.store.outputs.clearDestinationCooldown(this.destinationId); } catch { /* */ }
this.report({ state: 'live', live_at: Date.now(), last_error: null });
this.report({ state: 'live', live_at: Date.now(), last_error: null });
this.note('info', 'output is live');
this.note('info', 'output is live');
this.timer(() => {
this.timer(() => {
@@ -201,14 +201,14 @@ class OutputRunner {
@@ -201,14 +201,14 @@ class OutputRunner {
}
}
/** The circuit breaker: this output is done; the destination may cool down. */
/** The circuit breaker: this output is done; the destination may cool down. */
- fail(message, { cooldown }) {
+ async fail(message, { cooldown }) {
if (this.stopped) return;
if (this.stopped) return;
message = this.scrub(message);
message = this.scrub(message);
this.stopped = true;
this.stopped = true;
this.clearTimers();
this.clearTimers();
let cooldownMinutes = null;
let cooldownMinutes = null;
if (cooldown) {
if (cooldown) {
- const r = this.store.outputs.markDestinationFailure(this.destinationId, message);
+ const r = await this.store.outputs.markDestinationFailure(this.destinationId, message);
cooldownMinutes = r ? r.cooldownMinutes : null;
cooldownMinutes = r ? r.cooldownMinutes : null;
}
}
this.status = 'failed';
this.status = 'failed';
@@ -73,21 +73,22 @@ function createRtmpIngest({ rt, log = console, exit = (code) => process.exit(cod
@@ -73,21 +73,22 @@ function createRtmpIngest({ rt, log = console, exit = (code) => process.exit(cod
rt, kind: 'rtmp-ingest', log, exit,
rt, kind: 'rtmp-ingest', log, exit,
hooks: {
hooks: {
onDrain: () => closePublic(),
onDrain: () => closePublic(),
- onDrainDeadline: () => endAll('drain_deadline'),
- onEndRequested: (s) => endSession(s.id, 'end_requested'),
- onLost: () => { for (const [nmsId] of publishers) stopNms(nmsId); return closeAll(); },
- onHeartbeat: () => refreshMediaInfo(),
+ onDrainDeadline: async () => await endAll('drain_deadline'),
+ onEndRequested: async (s) => await endSession(s.id, 'end_requested'),
+ onLost: async () => { for (const [nmsId] of publishers) await stopNms(nmsId); return closeAll(); },
+ onHeartbeat: async () => await refreshMediaInfo(),
activeCount: () => publishers.size,
activeCount: () => publishers.size,
onExit: () => closeAll(),
onExit: () => closeAll(),
},
},
});
});
+ // node-media-server's own session registry (in memory), not the store.
function nms(id) { return context.sessions.get(id); }
function nms(id) { return context.sessions.get(id); }
function stopNms(id) { const s = nms(id); if (s) { try { s.reject(); } catch { /* socket already gone */ } } }
function stopNms(id) { const s = nms(id); if (s) { try { s.reject(); } catch { /* socket already gone */ } } }
function on(event, fn) {
function on(event, fn) {
- const wrapped = (...args) => {
- try { fn(...args); } catch (err) {
+ const wrapped = async (...args) => {
+ try { await fn(...args); } catch (err) {
// A throw inside a node-media-server event is an uncaught exception, and that would
// A throw inside a node-media-server event is an uncaught exception, and that would
// take every publisher on this worker with it.
// take every publisher on this worker with it.
log.error(`[rtmp] ${event} handler failed: ${err.stack || err}`);
log.error(`[rtmp] ${event} handler failed: ${err.stack || err}`);
@@ -101,50 +102,84 @@ function createRtmpIngest({ rt, log = console, exit = (code) => process.exit(cod
@@ -101,50 +102,84 @@ function createRtmpIngest({ rt, log = console, exit = (code) => process.exit(cod
return Boolean(session && session.socket && publicPorts.has(session.socket.localPort));
return Boolean(session && session.socket && publicPorts.has(session.socket.localPort));
}
}
- on('prePublish', (id, streamPath) => {
+ /**
+ * Admission of a publish: the key and the one-open-session rule are read from the store (PostgreSQL, async), but
+ * node-media-server's onPublish is synchronous and registers the stream right after 'prePublish'. So every RTMP
+ * session's publish command waits here first (newRtmpServer wraps onPublish), and 'prePublish' consumes the answer.
+ */
+ const admissions = new Map();
+ async function admit(s, streamPath) {
+ if (!isPublicSocket(s)) return { error: 'not the public listener' };
+ if (runtime.draining || !runtime.me || runtime.me.state !== 'ready') return { error: 'generation not taking sessions' };
+ if (publishers.size >= config.rtmp.maxPublishersPerWorker) return { error: 'worker full' };
+ const parts = String(streamPath).split('/');
+ if (parts.length !== 3 || parts[1] !== 'live') return { error: 'bad path' };
+ const r = await store.definitions.resolveIngestKey(parts[2], 'rtmp');
+ if (r.error) return { error: r.error };
+ const a = await store.sessions.admit({ definition: r.definition, key: r.key, protocol: 'rtmp', worker: runtime.me });
+ if (a.error) return { error: a.error };
+ return { sessionId: a.session.id, definitionId: r.definition.id, hint: r.key.hint };
+ }
+ function admitThenPublish(session) {
+ const onPublish = session.onPublish.bind(session);
+ session.onPublish = (invokeMessage) => {
+ // A second publish on a connection that already publishes: node-media-server answers
+ // NetStream.Publish.BadConnection without touching the first stream.
+ if (publishers.has(session.id)) return onPublish(invokeMessage);
+ const streamPath = `/${session.appname}/${String(invokeMessage.streamName || '').split('?')[0]}`;
+ admit(session, streamPath).then(async (adm) => {
+ if (!session.isStarting) {
+ // The publisher left while it was being admitted: its session ends without going live.
+ if (adm.sessionId) await store.sessions.finish(adm.sessionId, { reason: 'publisher_disconnected', actor: `worker:${runtime.me.id}` });
+ return;
+ }
+ admissions.set(session.id, adm);
+ onPublish(invokeMessage);
+ }, (err) => {
+ log.error(`[rtmp] admission failed: ${err.stack || err}`);
+ admissions.set(session.id, { error: 'admission failed' });
+ if (session.isStarting) onPublish(invokeMessage);
+ });
+ };
+ }
+
+ on('prePublish', (id) => {
const s = nms(id);
const s = nms(id);
if (!s) return;
if (!s) return;
const reject = (why) => { log.log(`[rtmp] publish refused (${why}) from ${s.ip}`); s.reject(); };
const reject = (why) => { log.log(`[rtmp] publish refused (${why}) from ${s.ip}`); s.reject(); };
- // A second publish on a connection that already publishes: leave it to node-media-server,
- // which answers NetStream.Publish.BadConnection without touching the first stream.
if (publishers.has(id)) return undefined;
if (publishers.has(id)) return undefined;
- if (!isPublicSocket(s)) return reject('not the public listener');
- if (runtime.draining || !runtime.me || runtime.me.state !== 'ready') return reject('generation not taking sessions');
- if (publishers.size >= config.rtmp.maxPublishersPerWorker) return reject('worker full');
- const parts = String(streamPath).split('/');
- if (parts.length !== 3 || parts[1] !== 'live') return reject('bad path');
- const r = store.definitions.resolveIngestKey(parts[2], 'rtmp');
- if (r.error) return reject(r.error);
- const a = store.sessions.admit({ definition: r.definition, key: r.key, protocol: 'rtmp', worker: runtime.me });
- if (a.error) return reject(a.error);
+ const adm = admissions.get(id);
+ admissions.delete(id);
+ if (!adm) return reject('not admitted');
+ if (adm.error) return reject(adm.error);
// Rename before node-media-server registers the publisher (it reads publishStreamPath
// Rename before node-media-server registers the publisher (it reads publishStreamPath
// right after this synchronous event): from here on the stream is /live/<session id>.
// right after this synchronous event): from here on the stream is /live/<session id>.
- s.publishStreamPath = `/live/${a.session.id}`;
- publishers.set(id, { sessionId: a.session.id, definitionId: r.definition.id, endReason: null });
- log.log(`[rtmp] session ${a.session.id} accepted for ${r.definition.id} (key …${r.key.hint})`);
+ s.publishStreamPath = `/live/${adm.sessionId}`;
+ publishers.set(id, { sessionId: adm.sessionId, definitionId: adm.definitionId, endReason: null });
+ log.log(`[rtmp] session ${adm.sessionId} accepted for ${adm.definitionId} (key …${adm.hint})`);
});
});
- on('postPublish', (id) => {
+ on('postPublish', async (id) => {
const p = publishers.get(id);
const p = publishers.get(id);
if (!p) return;
if (!p) return;
- const t = store.sessions.transition(p.sessionId, 'live', { reason: 'media_flowing', actor: `worker:${runtime.me.id}` });
+ const t = await store.sessions.transition(p.sessionId, 'live', { reason: 'media_flowing', actor: `worker:${runtime.me.id}` });
if (!t.ok) {
if (!t.ok) {
// The coordinator failed it meanwhile (lease) — the transport must not outlive its record.
// The coordinator failed it meanwhile (lease) — the transport must not outlive its record.
log.warn(`[rtmp] session ${p.sessionId} could not go live (${t.code}); closing`);
log.warn(`[rtmp] session ${p.sessionId} could not go live (${t.code}); closing`);
- stopNms(id);
+ await stopNms(id);
}
}
});
});
- on('donePublish', (id) => {
+ on('donePublish', async (id) => {
const p = publishers.get(id);
const p = publishers.get(id);
if (!p) return;
if (!p) return;
publishers.delete(id);
publishers.delete(id);
- store.sessions.finish(p.sessionId, { reason: p.endReason || 'publisher_disconnected', actor: `worker:${runtime.me.id}` });
+ await store.sessions.finish(p.sessionId, { reason: p.endReason || 'publisher_disconnected', actor: `worker:${runtime.me.id}` });
log.log(`[rtmp] session ${p.sessionId} ended (${p.endReason || 'publisher_disconnected'})`);
log.log(`[rtmp] session ${p.sessionId} ended (${p.endReason || 'publisher_disconnected'})`);
if (runtime.draining) runtime.exitWhenIdle();
if (runtime.draining) runtime.exitWhenIdle();
});
});
- on('prePlay', (id, streamPath) => {
+ on('prePlay', async (id, streamPath) => {
const s = nms(id);
const s = nms(id);
if (!s) return;
if (!s) return;
// RTMP play only on the loopback play port, and only by session path.
// RTMP play only on the loopback play port, and only by session path.
@@ -152,11 +187,11 @@ function createRtmpIngest({ rt, log = console, exit = (code) => process.exit(cod
@@ -152,11 +187,11 @@ function createRtmpIngest({ rt, log = console, exit = (code) => process.exit(cod
if (!SESSION_PATH_RE.test(String(streamPath))) s.reject();
if (!SESSION_PATH_RE.test(String(streamPath))) s.reject();
});
});
- function refreshMediaInfo() {
+ async function refreshMediaInfo() {
for (const [id, p] of publishers) {
for (const [id, p] of publishers) {
const s = nms(id);
const s = nms(id);
if (!s || !s.isPublishing) continue;
if (!s || !s.isPublishing) continue;
- store.sessions.setMediaInfo(p.sessionId, {
+ await store.sessions.setMediaInfo(p.sessionId, {
video_codec: s.videoCodecName || null,
video_codec: s.videoCodecName || null,
width: s.videoWidth || null,
width: s.videoWidth || null,
height: s.videoHeight || null,
height: s.videoHeight || null,
@@ -168,17 +203,17 @@ function createRtmpIngest({ rt, log = console, exit = (code) => process.exit(cod
@@ -168,17 +203,17 @@ function createRtmpIngest({ rt, log = console, exit = (code) => process.exit(cod
}
}
}
}
- function endSession(sessionId, reason) {
+ async function endSession(sessionId, reason) {
for (const [id, p] of publishers) {
for (const [id, p] of publishers) {
- if (p.sessionId === sessionId) { p.endReason = reason; stopNms(id); return true; }
+ if (p.sessionId === sessionId) { p.endReason = reason; await stopNms(id); return true; }
}
}
// Not ours any more (publisher already gone): make sure the record is closed.
// Not ours any more (publisher already gone): make sure the record is closed.
- store.sessions.finish(sessionId, { reason, actor: `worker:${runtime.me.id}` });
+ await store.sessions.finish(sessionId, { reason, actor: `worker:${runtime.me.id}` });
return false;
return false;
}
}
- function endAll(reason) {
- for (const [id, p] of publishers) { p.endReason = reason; stopNms(id); }
+ async function endAll(reason) {
+ for (const [id, p] of publishers) { p.endReason = reason; await stopNms(id); }
}
}
function closePublic() {
function closePublic() {
@@ -209,6 +244,7 @@ function createRtmpIngest({ rt, log = console, exit = (code) => process.exit(cod
@@ -209,6 +244,7 @@ function createRtmpIngest({ rt, log = console, exit = (code) => process.exit(cod
srv.openSockets.add(socket);
srv.openSockets.add(socket);
socket.on('close', () => srv.openSockets.delete(socket));
socket.on('close', () => srv.openSockets.delete(socket));
const session = new NodeRtmpSession(nmsConfig, socket);
const session = new NodeRtmpSession(nmsConfig, socket);
+ admitThenPublish(session);
session.run();
session.run();
});
});
srv.openSockets = new Set();
srv.openSockets = new Set();
@@ -233,7 +269,7 @@ function createRtmpIngest({ rt, log = console, exit = (code) => process.exit(cod
@@ -233,7 +269,7 @@ function createRtmpIngest({ rt, log = console, exit = (code) => process.exit(cod
});
});
const flvPort = await listenInRange(flvServer, config.rtmp.internalPortMin, config.rtmp.internalPortMax, new Set([rtmpPlayPort]));
const flvPort = await listenInRange(flvServer, config.rtmp.internalPortMin, config.rtmp.internalPortMax, new Set([rtmpPlayPort]));
endpoints = { publicPort: config.rtmp.port, publicPorts: [...publicPorts], rtmpPlayPort, flvPort };
endpoints = { publicPort: config.rtmp.port, publicPorts: [...publicPorts], rtmpPlayPort, flvPort };
- runtime.register(endpoints);
+ await runtime.register(endpoints);
for (const port of publicPorts) {
for (const port of publicPorts) {
const srv = newRtmpServer();
const srv = newRtmpServer();
@@ -244,7 +280,7 @@ function createRtmpIngest({ rt, log = console, exit = (code) => process.exit(cod
@@ -244,7 +280,7 @@ function createRtmpIngest({ rt, log = console, exit = (code) => process.exit(cod
srv.on('error', (err) => log.error(`[rtmp] public listener ${port}: ${err.message}`));
srv.on('error', (err) => log.error(`[rtmp] public listener ${port}: ${err.message}`));
publicServers.push(srv);
publicServers.push(srv);
}
}
- runtime.ready();
+ await runtime.ready();
log.log(`[rtmp] generation ${runtime.me.generation} ready: publish ${config.rtmp.bindHost}:${[...publicPorts].join(',')}, play 127.0.0.1:${rtmpPlayPort}, flv 127.0.0.1:${flvPort}`);
log.log(`[rtmp] generation ${runtime.me.generation} ready: publish ${config.rtmp.bindHost}:${[...publicPorts].join(',')}, play 127.0.0.1:${rtmpPlayPort}, flv 127.0.0.1:${flvPort}`);
return endpoints;
return endpoints;
}
}
@@ -254,29 +290,31 @@ function createRtmpIngest({ rt, log = console, exit = (code) => process.exit(cod
@@ -254,29 +290,31 @@ function createRtmpIngest({ rt, log = console, exit = (code) => process.exit(cod
runtime,
runtime,
endpoints: () => endpoints,
endpoints: () => endpoints,
publishers: () => [...publishers.values()],
publishers: () => [...publishers.values()],
- drain: (reason) => runtime.drainNow(reason),
+ drain: async (reason) => await runtime.drainNow(reason),
close: closeAll,
close: closeAll,
};
};
}
}
if (require.main === module) {
if (require.main === module) {
- require('dotenv').config({ path: process.env.OPENRE_ENV_FILE || path.join(process.cwd(), '.env') });
- const { load, exitIfDrill } = require('../server/config');
- const { openRuntime } = require('../server/store');
- const config = load();
- exitIfDrill(config, 'openre-rtmp-ingest');
- const rt = openRuntime({ config });
- const worker = createRtmpIngest({ rt });
- let started = false;
- worker.start().then(() => { started = true; }, (err) => { console.error(`[rtmp] failed to start: ${err.stack || err}`); process.exit(1); });
- // node-media-server's own server logs an uncaught exception and keeps going (NodeMediaServer.run
- // registers exactly that). A malformed packet from one encoder must not take every other
- // publisher on this worker down with it, so the worker keeps that behaviour once it is up.
- process.on('uncaughtException', (err) => {
- console.error(`[rtmp] uncaught exception (worker keeps running): ${err && err.stack || err}`);
- if (!started) process.exit(1);
- });
- for (const sig of ['SIGTERM', 'SIGINT']) process.on(sig, () => worker.drain(sig));
+ (async () => {
+ require('dotenv').config({ path: process.env.OPENRE_ENV_FILE || path.join(process.cwd(), '.env') });
+ const { load, exitIfDrill } = require('../server/config');
+ const { openRuntime } = require('../server/store');
+ const config = load();
+ exitIfDrill(config, 'openre-rtmp-ingest');
+ const rt = await openRuntime({ config });
+ const worker = createRtmpIngest({ rt });
+ let started = false;
+ worker.start().then(() => { started = true; }, (err) => { console.error(`[rtmp] failed to start: ${err.stack || err}`); process.exit(1); });
+ // node-media-server's own server logs an uncaught exception and keeps going (NodeMediaServer.run
+ // registers exactly that). A malformed packet from one encoder must not take every other
+ // publisher on this worker down with it, so the worker keeps that behaviour once it is up.
+ process.on('uncaughtException', (err) => {
+ console.error(`[rtmp] uncaught exception (worker keeps running): ${err && err.stack || err}`);
+ if (!started) process.exit(1);
+ });
+ for (const sig of ['SIGTERM', 'SIGINT']) process.on(sig, () => worker.drain(sig));
+ })().catch((err) => { console.error(`[rtmp-ingest] failed to start: ${err && err.stack || err}`); process.exit(1); });
}
}
module.exports = { createRtmpIngest, listenInRange };
module.exports = { createRtmpIngest, listenInRange };
@@ -37,22 +37,22 @@ function createWorkerRuntime({ rt, kind, log = console, hooks = {}, exit = (code
@@ -37,22 +37,22 @@ function createWorkerRuntime({ rt, kind, log = console, hooks = {}, exit = (code
let deadlineHandled = false;
let deadlineHandled = false;
let stopped = false;
let stopped = false;
- function register(endpoints = {}) {
- me = store.workers.register({ kind, endpoints });
+ async function register(endpoints = {}) {
+ me = await store.workers.register({ kind, endpoints });
log.log(`[${kind}] registered ${me.id} generation ${me.generation} (release ${me.release})`);
log.log(`[${kind}] registered ${me.id} generation ${me.generation} (release ${me.release})`);
return me;
return me;
}
}
- function ready() {
- store.workers.ready(me.id);
- me = store.workers.get(me.id);
+ async function ready() {
+ await store.workers.ready(me.id);
+ me = await store.workers.get(me.id);
schedule();
schedule();
return me;
return me;
}
}
- function beat() {
+ async function beat() {
if (stopped || !me) return null;
if (stopped || !me) return null;
- const row = store.workers.heartbeat(me.id);
+ const row = await store.workers.heartbeat(me.id);
me = row;
me = row;
if (!row) return null;
if (!row) return null;
if (row.state === 'lost') {
if (row.state === 'lost') {
@@ -73,7 +73,7 @@ function createWorkerRuntime({ rt, kind, log = console, hooks = {}, exit = (code
@@ -73,7 +73,7 @@ function createWorkerRuntime({ rt, kind, log = console, hooks = {}, exit = (code
try { hooks.onDrainDeadline && hooks.onDrainDeadline(row); } catch (err) { log.error(`[${kind}] onDrainDeadline: ${err.message}`); }
try { hooks.onDrainDeadline && hooks.onDrainDeadline(row); } catch (err) { log.error(`[${kind}] onDrainDeadline: ${err.message}`); }
}
}
if (hooks.onEndRequested) {
if (hooks.onEndRequested) {
- for (const s of store.sessions.endRequestsFor(me.id)) {
+ for (const s of await store.sessions.endRequestsFor(me.id)) {
try { hooks.onEndRequested(s); } catch (err) { log.error(`[${kind}] onEndRequested: ${err.message}`); }
try { hooks.onEndRequested(s); } catch (err) { log.error(`[${kind}] onEndRequested: ${err.message}`); }
}
}
}
}
@@ -87,20 +87,20 @@ function createWorkerRuntime({ rt, kind, log = console, hooks = {}, exit = (code
@@ -87,20 +87,20 @@ function createWorkerRuntime({ rt, kind, log = console, hooks = {}, exit = (code
function schedule() {
function schedule() {
clearTimeout(timer);
clearTimeout(timer);
if (stopped) return;
if (stopped) return;
- timer = setTimeout(() => {
- try { beat(); } catch (err) { log.error(`[${kind}] heartbeat failed: ${err.message}`); }
+ timer = setTimeout(async () => {
+ try { await beat(); } catch (err) { log.error(`[${kind}] heartbeat failed: ${err.message}`); }
schedule();
schedule();
}, config.workers.heartbeatMs);
}, config.workers.heartbeatMs);
}
}
/** SIGTERM: the operator (or systemd) asks this generation to go away → same as a drain. */
/** SIGTERM: the operator (or systemd) asks this generation to go away → same as a drain. */
- function drainNow(reason = 'signal') {
+ async function drainNow(reason = 'signal') {
if (!me || stopped) return;
if (!me || stopped) return;
if (!draining) {
if (!draining) {
- store.workers.drain(me.id, Date.now() + config.workers.drainMaxMs);
+ await store.workers.drain(me.id, Date.now() + config.workers.drainMaxMs);
log.log(`[${kind}] draining on ${reason}`);
log.log(`[${kind}] draining on ${reason}`);
}
}
- beat();
+ await beat();
}
}
function exitWhenIdle() {
function exitWhenIdle() {
@@ -1,3 +1,3 @@
@@ -1,3 +1,3 @@
'use strict';
'use strict';
/** openre-sfu: worker interface only, the transport is not ported yet (see unported-transport.js). */
/** openre-sfu: worker interface only, the transport is not ported yet (see unported-transport.js). */
-if (require.main === module) require('./unported-transport').main('sfu');
+if (require.main === module) require('./unported-transport').main('sfu').catch((err) => { console.error(`[sfu] failed to start: ${err && err.stack || err}`); process.exit(1); });
@@ -24,23 +24,23 @@ function createUnportedTransport({ rt, kind, log = console, exit = (code) => pro
@@ -24,23 +24,23 @@ function createUnportedTransport({ rt, kind, log = console, exit = (code) => pro
const runtime = createWorkerRuntime({ rt, kind, log, exit, hooks: { activeCount: () => 0 } });
const runtime = createWorkerRuntime({ rt, kind, log, exit, hooks: { activeCount: () => 0 } });
return {
return {
runtime,
runtime,
- start() {
- runtime.register({ ported: false, note: PORT_STATUS[kind] });
- runtime.ready();
+ async start() {
+ await runtime.register({ ported: false, note: PORT_STATUS[kind] });
+ await runtime.ready();
log.log(`[${kind}] registered generation ${runtime.me.generation}: ${PORT_STATUS[kind]}`);
log.log(`[${kind}] registered generation ${runtime.me.generation}: ${PORT_STATUS[kind]}`);
return runtime.me;
return runtime.me;
},
},
- drain: (reason) => runtime.drainNow(reason),
+ drain: async (reason) => await runtime.drainNow(reason),
};
};
}
}
-function main(kind) {
+async function main(kind) {
require('dotenv').config({ path: process.env.OPENRE_ENV_FILE || path.join(process.cwd(), '.env') });
require('dotenv').config({ path: process.env.OPENRE_ENV_FILE || path.join(process.cwd(), '.env') });
const { load, exitIfDrill } = require('../server/config');
const { load, exitIfDrill } = require('../server/config');
const { openRuntime } = require('../server/store');
const { openRuntime } = require('../server/store');
const config = load();
const config = load();
exitIfDrill(config, `openre-${kind}`);
exitIfDrill(config, `openre-${kind}`);
- const w = createUnportedTransport({ rt: openRuntime({ config }), kind });
+ const w = createUnportedTransport({ rt: await openRuntime({ config }), kind });
w.start();
w.start();
for (const sig of ['SIGTERM', 'SIGINT']) process.on(sig, () => w.drain(sig));
for (const sig of ['SIGTERM', 'SIGINT']) process.on(sig, () => w.drain(sig));
}
}
@@ -1,3 +1,3 @@
@@ -1,3 +1,3 @@
'use strict';
'use strict';
/** openre-webrtc-ingest: worker interface only, the transport is not ported yet (see unported-transport.js). */
/** openre-webrtc-ingest: worker interface only, the transport is not ported yet (see unported-transport.js). */
-if (require.main === module) require('./unported-transport').main('webrtc-ingest');
+if (require.main === module) require('./unported-transport').main('webrtc-ingest').catch((err) => { console.error(`[webrtc-ingest] failed to start: ${err && err.stack || err}`); process.exit(1); });